adriaan,

Multi-process solution in #Perl for the 1 billion record challenge. Runs in about 4 minutes with 8 child processes compared to 30 minutes with the baseline Perl solution (on my hardware). https://github.com/adriaandens/1brc.pl

There's still a lot of room for further optimalizations.

There's an ongoing discussion on the Perl Reddit about the 1BRC. One cool trick someone did was 'use integer;' and work without the default float behaviour of the numbers to get a 10% speed boost. https://www.reddit.com/r/perl/comments/18ygpsi/1_billion_row_challenge_in_perl/

@Perl

mjgardner, (edited )
@mjgardner@social.sdf.org avatar

@adriaan @Perl Per @karjala on #Perl Reddit (https://www.reddit.com/r/perl/comments/18ygpsi/comment/kgf8f49/?context=3), have you tried #CPAN modules like https://metacpan.org/pod/Parallel::ForkManager or (my weapon of choice) https://metacpan.org/dist/MCE/view/lib/MCE.pod ?

adriaan,

@mjgardner @Perl Parallel::ForkManager was easy to implement from what I've made. I'll have to look into MCE to know which interface is best for the 1BRC problem and which also gives the needed control (mostly being able to control memory usage since I might not be able to slurp the 13GB if there's hidden overhead (my pc has 16GB total))

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • Perl@chirp.social
  • ngwrru68w68
  • rosin
  • GTA5RPClips
  • osvaldo12
  • love
  • Youngstown
  • slotface
  • khanakhh
  • everett
  • kavyap
  • mdbf
  • DreamBathrooms
  • thenastyranch
  • magazineikmin
  • megavids
  • InstantRegret
  • normalnudes
  • tacticalgear
  • cubers
  • ethstaker
  • modclub
  • cisconetworking
  • Durango
  • anitta
  • Leos
  • tester
  • provamag3
  • JUstTest
  • All magazines
    •