The sparklyr package and friends have been getting some important updates in the past few months!
sparklyr is a package that allows you to interact with Spark using familiar R interfaces, such as dplyr, broom, and DBI. You can also gain access to Spark's distributed Machine Learning libraries, Structure Streaming, and ML Pipelines from R.
Pysparklyr is the new extension to sparklyr that allows you to interact with #Spark & Databricks Connect. The new version has big user-facing updates that make working with #Databricks and #RStats together even easier.
We are thrilled to announce that the latest version of sparklyr is on CRAN. sparklyr is the popular and powerful #RStats interface for #Apache#Spark, including Spark clusters hosted in #Databricks.
Thanks to the new Spark Connect protocol, you can access Spark’s powerful distributed computing features from RStudio Desktop, a Posit Workbench instance, or any running R terminal or process.
i wish #spark had native UDFs, or whatever you’d call this:
@native_udf
def add_n(c: Column: n: Column) -> Column:
return c + n
you can easily make #python or #scala functions like that today, but you can’t reference them from #sql. i prefer them when possible in python bc they run natively. even in scala, strings aren’t converted to utf16 so it’s a win there too. they’re just a bit awkward when improving large chunks of SQL
My 6-yr old asked one of the BIG questions the other day.
"Daddy, why do we live?"
Knowing that he's being influenced by the ubiquitous #god botherers, I responded:
"Because of #electricity, little buddy. Kind of like the lightbulbs in our house, we all have a #spark in us that runs through our #bodies and #brains. When that spark goes out, we die."
He thought a bit about that, and then came the next BIG question:
"We become a part of the electricity that runs through the entire #planet that let's everything live. Which is super important, because without that electricity, nothing would live."
Various thoughts on too many programming languages, for no discernible reason.
I have been interested in Go since it's very initial release, but their dependence on Google is uncharming to say the least. I still haven't made up my mind on its GC, but its definitely better than most.
I used to do some ML work in .NET and if it wasn't dependent on Microsoft it would be a heavy contender for a great language, but it has far too many Microsoft-isms to ever really go much farther.
Rust is great, I enjoy beating my head against a brick wall battling with the compiler, and their safety is great, but overly complicated and feature-creep is a real problem on that entire project. I do a lot of work these days in Rust, for better (mostly) or worse (mostly-ish).
C is my bread-and-butter, as is Javascript for quick prototyping.
Elixir is great, but Erlang is unwieldy, the community is growing, but not fast enough - and I just can't get my mind to enjoy the syntax no matter how nice it is.
D is a lot of fun, but their GC can be slow at times, and the community is very small and packages are often broken and unmaintained.
Python was my first true love, but I really can't stand the whitespace, again love the language, hate the syntax.
Zig is fun, but just that. Fast, nimble, but early days, a bit confusing, could replace my insistence on C for core projects, but again, early days. I love to use them as a compiler for C, much faster than the defaults on any of the others.
Odin is one I love to keep an eye on, I wish I could get behind using it for more things. When I first took notice ~4 years ago the documentation was a bit scattered, but it looks much better now. The developer behind it is incredibly cool, could be seen as the next Dennis Ritchie imo. Runes are dope. The syntax is by far my favourite.
Julia, I love Julia, but performance last I tested was a bit of a miss, and by miss, it required a decent chunk of compute for basics, but when you gave it the system to throttle, it would be insanely productive to write in. Javascript is something that I prototype even syscalls in, but Julia is just the same but much better and more productive (and less strange) in many regards. I am really hoping this takes over in the ML/Data world and just eats Python alive. I've heard there has been major work in the perf department, but I haven't had reason to try it out lately.
Ada, memory safety before Rust! Great language, especially for critical applications, decades of baggage (or wisdom), slow moving language, insanely stable, compilers are all mostly proprietary, job market is small, but well paid, great for robotics, defense, and space industry types, but the syntax is... rough. Someone should make a meta-language on top of Ada like Zig/Nim/Odin do for C, or Elixir does for Erlang.
The others: Carbon, haven't tried; Nim, prefer when they were "Nimrod" (cue Green Day), decent but not my style; Crystal, seems cool, but not for me; Scala, great FP language, but JVM; Haskell, I'm not a mathematician, but my mathematician friends love it. I see why, but not my thing as much as I love functional languages. I'll try it again, eventually. I did not learn Haskell a great good.
I tend to jump from language to language, trying everything out, it's fun and a total timesuck.
Since writing this I've started looking more into Lisp, and I am starting to understand why everyone things its "gods chosen language". it's great, and you can turn/embed any language into a lisp. That's cool.
I like both Ada and C, but I work in the robotics industry, so we have to constantly switch between the two for anything hardware based. They both have their benefits, C is my preferred because of how simple (it can be, if you try...), but Ada (Primarily SPARK is what I've done work in) is great, but dated in many regards. I don't know if I've ever looked into Austral, but I'll take a look!
Haskell, tried it many times, never a good time. Ocaml isn't bad, but again, not my thing or style but definitely something I inherently understand more.
Scala, JVM, my sworn enemy. Never again. They can't get me suckered into reading Java docs again.
Elixir, great language, but I am going to agree with you - I am starting to prefer working with Erlang directly, but its early days on that. I've started looking into @lfe which looks incredible.
Nim, I dislike whitespace reqs in languages, I tend to value customization of my styles to make it all make sense to me, so Nim is too controlling for me in that regard. I read code a lot more than write it.
No real opinion on C2-3, love D, V looks interesting but not sure yet, Jai - if it ever comes out, haven't heard of Scopes, and I already mentioned Carbon.
If someone could find a way to package Rusts memory management in a tenable way to be cross-language, we would have a massive explosion of greatness. I know it's possible, but nearly impossible without some wicked genius' at the helm.
I'm looking for a role leading an engineering team. I've got several years both managing teams as well as an individual contributor. Strong recent #BigData and #DataEngineering experience.
lukewarm take (had this talk in private with @mia but I have to share this):
#Rust is the best thing to happen to #Ada and they should be used a lot more (especially Ada). Moreso, I believe that Rust is essentially FP Ada with C syntax.
Now let me cook (this will be a multi-toot series). Ada, like Rust, is extremely type safe and stringent (they reach memory safety through different paradigms, but they end up having equivalent levels of memory safety).
Ada is more procedural/OO while Rust is more functional, from that POV they fit together perfectly. They're both fast languages with a lot of checks.
Rust can benefit from what we Ada people have with #SPARK and actually be able to prove that your program does what it says without a shadow of a doubt (SPARK is so good that it overshadows #MISRA C which is the gold standard for critical software made in C).
Ada can use Rust's popularity in open source and community (ideally not a toxic one at times), Rust can use Ada's proven track record of handling mission-critical tasks with no compromises whatsoever.
Ada can use Rust's successful marketing (holy shit does #AdaCore need to get involved more), Rust can use the lessons that Ada learned in its 40+ years of existence and improve upon them.
Rust can definitely benefit from having 1. an actual specification and 2. getting rustc certified (which would mean LLVM by extension), but those are behemoths and it's extremely unlikely, impossible even to audit. Lack of advertising leads to misconceptions, and misconceptions lead to not using the language (I use Tcl and Perl, so I definitely know how it feels to use languages that have been osbourned for 2+ decades).
I hate how the Gmail app spies on you, so I started looking for another app. I've been using Spark for a while now and I really like it.
I even started using the desktop app. Haven't used a desktop app for mail in ages.
Now if I could switch away from Gmail completely, that would be dandy.
ICYMI: @Posit and Databricks are partnering to provide #RStats and #Python developers with an easier way to connect workspaces, access data, run code, and interact with other resources. 🎉
What does this entail?
• Enhanced access to your Databricks Workspace from Posit Workbench
• Support for #Spark Connect in R via sparklyr
• Posit Workbench in Databricks Marketplace
@evan Bravo Ashley Fraser from the #CBC !! Finally, someone that explained the "complexity" of the #fediverse that is easy to digest for the wider public:)
I especially liked how she explained that #mastodon was ONE of the most popular services available.
About 3 years ago, I reached out to #spark the podcast on CBC about covering this rising social network, and they didn't seem to think it was noteworthy ... not sure if they've covered it since.