I need to read a large text file of around 5-6 GB line by line using Java.
How can I do this quickly?
Friday, 30 March 2018
c++ - Why is my program slow when looping over exactly 8192 elements?
The following tests have been done with Visual C++ compiler as it is used by the default Qt Creator install (I guess with no optimization flag). When using GCC, there is no big difference between Mystical's version and my "optimized" code. So the conclusion is that compiler optimizations take care off micro optimization better than humans (me at last). I leave the rest of my answer for reference.
It's not efficient to process images this way. It's better to use single dimension arrays. Processing all pixels is the done in one loop. Random access to points could be done using:
pointer + (x + y*width)*(sizeOfOnePixel)
In this particular case, it's better to compute and cache the sum of three pixels groups horizontally because they are used three times each.
I've done some tests and I think it's worth sharing. Each result is an average of five tests.
Original code by user1615209:
8193: 4392 ms
8192: 9570 ms
Mystical's version:
8193: 2393 ms
8192: 2190 ms
Two pass using a 1D array: first pass for horizontal sums, second for vertical sum and average.
Two pass addressing with three pointers and only increments like this:
imgPointer1 = &avg1[0][0];
imgPointer2 = &avg1[0][SIZE];
imgPointer3 = &avg1[0][SIZE+SIZE];
for(i=SIZE;i
}
8193: 938 ms
8192: 974 ms
Two pass using a 1D array and addressing like this:
for(i=SIZE;i
}
8193: 932 ms
8192: 925 ms
One pass caching horizontal sums just one row ahead so they stay in cache:
// Horizontal sums for the first two lines
for(i=1;i
}
// Rest of the computation
for(;i
hsumPointer[i]=imgPointer[i-1]+imgPointer[i]+imgPointer[i+1];
// Final result
resPointer[i-SIZE]=(hsumPointer[i-SIZE-SIZE]+hsumPointer[i-SIZE]+hsumPointer[i])/9;
}
8193: 599 ms
8192: 652 ms
Conclusion:
- No benefits of using several pointers and just increments (I thought it would have been faster)
- Caching horizontal sums is better than computing them several time.
- Two pass is not three times faster, two times only.
- It's possible to achieve 3.6 times faster using both a single pass and caching an intermediary result
I'm sure it's possible to do much better.
NOTE
Please, note that I wrote this answer to target general performance issues rather than the cache problem explained in Mystical's excellent answer. At the beginning it was just pseudo code. I was asked to do tests in the comments... Here is a completely refactored version with tests.
character - Is there a special significance to the cat in The Closer? - Movies & TV
When Deputy Chief Brenda Leigh Johnson moves into a new house in season one (two?), she discovers a cat that has been living in the home already, whom she names "Kitty".
"Kitty" is a she, but Brenda keeps referring to her pet as a "he," despite being constantly corrected by her significant other. While her initial ambivalence towards the cat changes, this incorrect gender identification does not.
In the beginning of season five, the cat passes away. She grieves and carries the cat's ashes around with her, first as a mistake, and then as a remembrance. The ashes are finally placed on the mantle in a fancy urn, despite Brenda's opposition to such practices earlier.
In literature, cats carry a symbolic value, for things like cleanliness and good fortune (see Wikipedia). It seems as though for being such a minor "character" this cat gets a lot of attention in the series to not mean something to the plot or character development.
Is there a deeper connection between Brenda and the cat that the writers and director are trying to establish? Are they comparing Brenda's strong skills in interrogation to "catching a rat"? If she's so close with the cat, why does she maintain this distance with the name "Kitty" and referring to she as a "he"? (drawing on any hints from official sources or evaluations from critics if possible)
Answer
In this interview with executive producer James Duff, he says:
Though Brenda is not a "pet person" she adopts the cat, simply calling
it "kitty," and begins to develop a relationship with her. "A pet is
usually not a part of the procedural aspects of a crime show, but it
is a part of everyday life for a lot of people and we wanted to show
Brenda as an ordinary person," says Duff.
As for the death of kitty:
Kyra Sedgwick broke down weeping when she arrived on set to begin
filming the fifth season of TNT's hit show "The Closer" and learned
that her character, Deputy Police Chief Brenda Leigh Johnson, would be
grappling in ensuing episodes with the serious illness of Kitty, her
pet cat. Unbeknownst to the show's creator and executive producer,
James Duff, Sedgwick had just lost her real-life cat.
"We didn't feel like we had a choice," says Duff, explaining why he
decided to make Kitty's illness a major storyline. The cat that played
Kitty had, in real life, been diagnosed with a serious kidney
condition. "We could replace the cat and keep filming, or we could
accept the cat was seriously ill and write it into the story."
PHP and Enumerations
I know that PHP doesn't have native Enumerations. But I have become accustomed to them from the Java world. I would love to use enums as a way to give predefined values which IDEs' auto-completion features could understand.
Constants do the trick, but there's the namespace collision problem and (or actually because) they're global. Arrays don't have the namespace problem, but they're too vague, they can be overwritten at runtime and IDEs rarely (never?) know how to autofill their keys.
Are there any solutions/workarounds you commonly use? Does anyone recall whether the PHP guys have had any thoughts or decisions around enums?
Answer
Depending upon use case, I would normally use something simple like the following:
abstract class DaysOfWeek
{
const Sunday = 0;
const Monday = 1;
// etc.
}
$today = DaysOfWeek::Sunday;
However, other use cases may require more validation of constants and values. Based on the comments below about reflection, and a few other notes, here's an expanded example which may better serve a much wider range of cases:
abstract class BasicEnum {
private static $constCacheArray = NULL;
private static function getConstants() {
if (self::$constCacheArray == NULL) {
self::$constCacheArray = [];
}
$calledClass = get_called_class();
if (!array_key_exists($calledClass, self::$constCacheArray)) {
$reflect = new ReflectionClass($calledClass);
self::$constCacheArray[$calledClass] = $reflect->getConstants();
}
return self::$constCacheArray[$calledClass];
}
public static function isValidName($name, $strict = false) {
$constants = self::getConstants();
if ($strict) {
return array_key_exists($name, $constants);
}
$keys = array_map('strtolower', array_keys($constants));
return in_array(strtolower($name), $keys);
}
public static function isValidValue($value, $strict = true) {
$values = array_values(self::getConstants());
return in_array($value, $values, $strict);
}
}
By creating a simple enum class that extends BasicEnum, you now have the ability to use methods thusly for simple input validation:
abstract class DaysOfWeek extends BasicEnum {
const Sunday = 0;
const Monday = 1;
const Tuesday = 2;
const Wednesday = 3;
const Thursday = 4;
const Friday = 5;
const Saturday = 6;
}
DaysOfWeek::isValidName('Humpday'); // false
DaysOfWeek::isValidName('Monday'); // true
DaysOfWeek::isValidName('monday'); // true
DaysOfWeek::isValidName('monday', $strict = true); // false
DaysOfWeek::isValidName(0); // false
DaysOfWeek::isValidValue(0); // true
DaysOfWeek::isValidValue(5); // true
DaysOfWeek::isValidValue(7); // false
DaysOfWeek::isValidValue('Friday'); // false
As a side note, any time I use reflection at least once on a static/const class where the data won't change (such as in an enum), I cache the results of those reflection calls, since using fresh reflection objects each time will eventually have a noticeable performance impact (Stored in an assocciative array for multiple enums).
Now that most people have finally upgraded to at least 5.3, and SplEnum is available, that is certainly a viable option as well--as long as you don't mind the traditionally unintuitive notion of having actual enum instantiations throughout your codebase. In the above example, BasicEnum and DaysOfWeek cannot be instantiated at all, nor should they be.
How to use SharedPreferences in Android to store, fetch and edit values
I want to store a time value and need to retrieve and edit it. How can I use SharedPreferences to do this?
Answer
To obtain shared preferences, use the following method
In your activity:
SharedPreferences prefs = this.getSharedPreferences(
"com.example.app", Context.MODE_PRIVATE);
To read preferences:
String dateTimeKey = "com.example.app.datetime";
// use a default value using new Date()
long l = prefs.getLong(dateTimeKey, new Date().getTime());
To edit and save preferences
Date dt = getSomeDate();
prefs.edit().putLong(dateTimeKey, dt.getTime()).apply();
The android sdk's sample directory contains an example of retrieving and storing shared preferences. Its located in the:
/samples/android-/ApiDemos directory
Edit==>
I noticed, it is important to write difference between commit() and apply() here as well.
commit() return true if value saved successfully otherwise false. It save values to SharedPreferences synchronously.
apply() was added in 2.3 and doesn't return any value either on success or failure. It saves values to SharedPreferences immediately but starts an asynchronous commit.
More detail is here.
python - what does n += 2 in this def?
I have a def function looks like:
def s(xs, n, m):
t = []
while n < m:
t.append(xs[n])
n += 2
return t
I understand the above code by t.append(xs[n]), but have no idea what n += 2 means here.
Any help will be appreciated.
Thanks
json - Post an array of objects from an HTML form using Flask?
I want to generate this JSON object containing an array of objects from form inputs:
{
"network":[
{"layer_type": "conv2d", "num_filters": 16, "kernel_size": 2, "padding": "valid", "stride": 2},
{"layer_type": "max_pool2d", "num_filters": 16, "kernel_size": 2, "padding": "valid", "stride": 2},
{"layer_type": "conv2d", "num_filters": 32, "kernel_size": 3, "padding": "valid", "stride": 2}
]
}
Is there a way I can do this using Flask?
Update
Here's what the form looks like:
As for the snippet of html code dynamically generated:
Edit: Since this question is being marked as duplicate I'm going to add more info. I want to achieve something like this in this question but using Flask:
{"students" => [
{
"first" => "foo",
"last" => "bar",
"age" => "21"
},
{
"first" => "baz",
"last" => "qux",
"age" => "19"
}
]}
It does work with Ruby according to the accepted answer there by having this kind of form:
But I want to know how to do it using Flask.
Subscribe to:
Posts (Atom)