Lobrien

@Lobrien@hachyderm.io

A writer who programs. Formerly, a programmer who wrote. ML for Earth. Formerly: Azure ML, Xamarin, Gemini Observatory, magazines (AI Expert, Game Developer, Computer Language), etc. Fascinated by "how do devs learn?"

#ML #Hawaii #MLOps #Pytorch #MetricLearning #FastAI #MarineBio #TTRPG #Blackwater #UnderwaterPhotography Searchable on #tfr

This profile is from a federated server and may be incomplete. Browse more on the original instance.

Lobrien, to ML

This is a very nice video on understanding attention in transformers https://www.3blue1brown.com/lessons/attention

#ML

kellogh, to random
@kellogh@hachyderm.io avatar

i’ve seen a lot in my career, but until now, i’ve never seen religious fervor like there is against AI. there’s a lot of religion around AI in most directions, but crypto had that too

what’s crazy about this phase is that if people took time to understand what accelationists see, they wouldn’t come to the same conclusions, and we’d have far more interesting and productive conversations about it

but instead we’re caught in tired arguments about “it’s just linear algebra”

Lobrien,

@kellogh The real issues, even the most obvious ones of bias-in, bias-out and real-world harms, have been totally drowned out by competition for meme-worthy phrasings of specious nonsense. It’s dismaying.

Lobrien, to random

Why are they spreading “unpasteurized milk is fine, actually!” misinformation? Is it just they never met a gov’t health conspiracy they didn’t like? These idiots are going to facilitate human-human transmission of bird flu; who wants that? Seriously?

kellogh, to LLMs
@kellogh@hachyderm.io avatar

i’m very excited about the interpretability work that #anthropic has been doing with #LLMs.

in this paper, they used classical machine learning algorithms to discover concepts. if a concept like “golden gate bridge” is present in the text, then they discover the associated pattern of neuron activations.

this means that you can monitor LLM responses for concepts and behaviors, like “illicit behavior” or “fart jokes”

https://www.anthropic.com/research/mapping-mind-language-model

Lobrien,

@kellogh This does, of course, imply vastly easier subversion of guardrails. Bad actors will have an easier time manipulating bias.

Lobrien, to random

Nothing can be done to stop this, says only industry where this regularly happens. https://mastodon.world/@hn100/112493477923174205

kellogh, to LLMs
@kellogh@hachyderm.io avatar

if i had more time, i'd love to investigate PII coming from #LLMs. i've seen it generate phone numbers and secrets, but i wonder if these are real or not. i imagine you could look at the logits to figure out if phone number digits were randomly chosen or if the sequence is meaningful to the LLM. anyone aware of researchers who have already done this?

Lobrien,

@kellogh Someone claimed that a long magic number used in their highly-optimized (FFT?) code was spit out by Copilot. (This was soon after release.) The constant was arrived at by long fine-tuning, not conceptual in any way.

ramikrispin, to machinelearning
@ramikrispin@mstdn.social avatar

MLX Examples 🚀

The MLX is Apple's framework for machine learning applications on Apple silicon. The MLX examples repository provides a set of examples for using the MLX framework. This includes examples of:
✅ Text models such as transformer, Llama, Mistral, and Phi-2 models
✅ Image models such as Stable Diffusion
✅ Audio and speech recognition with OpenAI's Whisper
✅ Support for some Hugging Face models

🔗 https://github.com/ml-explore/mlx-examples

Lobrien,

@ramikrispin @BenjaminHan How do this and corenet (https://github.com/apple/corenet) fit together? The corenet repo has examples for inference with MLX for models trained with corenet; is that it, does MLX not have, e.g., activation and loss fns, optimizers, etc.?

Lobrien, to random

Incredibly interesting write-up on AI-assisted improvements, skill leveling, and a "jagged frontier," where falling asleep at the wheel is a crucial mistake

https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged

kellogh, to LLMs
@kellogh@hachyderm.io avatar

i used an analogy yesterday, that #LLMs are basically system 1 (from Thinking Fast and Slow), and system 2 doesn’t exist but we can kinda fake it by forcing the LLM to have an internal dialog.

my understanding is that system 1 was more tuned to pattern matching and “gut reactions”, while system 2 is more analytical

i think it probably works pretty well, but curious what others think

Lobrien,

@kellogh I use that exact analogy. And emphasize that we certainly do use and need System 2 at least occasionally. At some point, human-like reasoning must use symbols with definite, not probabilistic, outcomes. Can that arise implicitly within attention heads? Similar to embeddings being kinda-sorta knowledge representation? I mean, maybe? But it still seems hugely wasteful to me. I still tend towards neuro-symbolic being the way.

christianselig, to random
@christianselig@mastodon.social avatar

Learning some basic 3D modeling has been such a fun hobby. Needed something to hold tennis balls since mine came in a cardboard box, so I quickly put together something fun 😊 https://makerworld.com/en/models/445402

Lobrien,

@christianselig Wow, I'd love to be able to... uh... "hexellate"?... surfaces. Was that manual or a feature of what you used (or did you make that in OpenSCAD or something, which would be ultra-impressive)?

timbray, (edited ) to random
@timbray@cosocial.ca avatar

This apparently touched a nerve; I had no idea. Check the comment thread.

My own major gripe with passkeys is that I could never find a simple straightforward explanation of what they were and how they were to be used. I have a decent understanding of asymmetric crypto and PKI and key exchange and JWT and so on, so if you can’t explain it to me, you have a big problem.

https://cosocial.ca/@timbray/112339186728840038

Lobrien,

@growse I, and I'm sure others, would be (note I've taken tbray out of the response for now). But "straightforward" for technical folks who understand PK components and protocols.

Lobrien,

@growse Yes, that would be exactly the type of starting place that I think would be helpful.

Lobrien,

@growse Well, there you go! I'm not clear on details, but that's fine. Am I right to paraphrase as "It's exactly like the Yubikey process, but with an abstraction that allows you to swap the physical touch + PIN for a p/w mgr password or an OS biometric or what have you"?

Lobrien, to random

I have to say, I thought by MS-DOS 4.0 there'd be more C in the codebase, but it's almost all ASM.

https://github.com/microsoft/MS-DOS/tree/main/v4.0

Lobrien, to swift

@Migueldeicaza What did you use to write https://migueldeicaza.github.io/SwiftGodotDocs/tutorials/swiftgodot/your-first-extension ? I was thinking of doing Pong Wars to learn #SwiftGodot and writing up my experience. #Swift #SwiftDocs

Lobrien, to random

For traveling, I like to be able to order off the menu in the native language. I got sick of #Duolingo's gamification and tried #Pimsleur for French. That's what I'd use if I were spending significant time there: Pimsleur is hard but no nonsense. I'm now on #Babbel but it's even easier than Duolingo and doesn't feel like I'm advancing at all. I might go back to Duolingo and submit to the gamification. Thoughts?

Lobrien, to random

Whaaaatttt? Memories may be constructed by breaking and patching DNA?!?! https://www.nature.com/articles/d41586-024-00930-y

rakyll, to random

I don't know if AI is going to replace programmers or not but there will be a lot of jobs just to delete AI generated code.

Lobrien,

@rakyll @rwlowry A major part of “programming” is going to switch to a two-step process of “Is this what was wanted?” and, if so, refactoring it to fit better within the overall codebase. For writers, this has long been essential (“writing is rewriting”). For devs, it’s hard to predict difficulty, because while pundits have long emphasized “code is for humans, not computers,” many programmers (at least junior programmers) think their job is that first draft.

Lobrien, to ML

Prompt "engineering" boils my blood. Can you imagine if you were working on a stream prediction system and the quality of the output depended on prepending a stream of magic numbers? You'd disdain anyone claiming that was a sustainable solution for a business. (I mean, I can imagine it, because that's exactly the kind of crap you see in consulting.) #ML

khalidabuhakmeh, to python
@khalidabuhakmeh@mastodon.social avatar

I’m doing #python today...

screaming homer simpson GIF

Lobrien,

@khalidabuhakmeh @lukem Come for the awkward object-orientation, stay for the disastrous performance! (Were it not for the fact that it's absolutely won the library ecosystem race...)

Lobrien, to Nvidia

Sold 3/4 of my #NVidia stock this week. Kept 1/4 because of "who knows?" upside risk. But analytically, I can't see anyway they protect their current valuation. Yes, they are almost a monopoly today. And as much chance as anyone to be the market leader for years. But I don't see how they really become a long-term monopoly on "AI chips." There's too much money in the pot.

kellogh, to random
@kellogh@hachyderm.io avatar

why would you want a machine to be conscious? setting aside the debate of if it is, what would that let it do that an unconscious machine couldn't?

Lobrien,

@kellogh Penrose claims that mathematical certainty can only be done with consciousness and is crucial to reason. (I think he conflates “conscious certainty” w. “ability to short-circuit a long-running loop,” but he’s got a Nobel and I don’t, so…) Also, I can imagine a “propagating consciousness is inherently good” argument.

Lobrien,

@kellogh Putting aside the fact that he’s 100x smarter than me, I just have never felt compelled by his arguments. Why can’t there be a monitor of internal thoughts that is constantly looking to insert a cached “Yes, that’s true, skip ahead,” or “No, that’s false, prune this exploration”? It seems like exactly the sort of thing evolution would provide.

Lobrien, to ML

Any good sources on what the outputs of the attention blocks in a transformer represent? I expected that for "The bank of the plane took it around the savings bank on the bank of the river", the vectors corresponding to "bank" would diverge -- "rotation things/money things/rivery things" -- but AFAICT that doesn't clearly happen. Here are the dot prods of the normalized vectors (aka "cosine similarity") against themselves after embedding layer and attention block 5:

Heatmap showing identical vectors for identical word embeddings

Lobrien,

@kellogh Yeah, there’s a linear layer after the attention is applied. I more or less expected that to swizzle things up, which is why the continued correlation between “rotation bank” “money bank” “river bank” surprises. I thought they’d diverge (they don’t in a clear way) but if I swapped in “embankment of the river” that then some vector in the transformer-block output would converge with “river bank”. Haven’t done that code yet.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • megavids
  • mdbf
  • ngwrru68w68
  • modclub
  • magazineikmin
  • thenastyranch
  • rosin
  • khanakhh
  • InstantRegret
  • Youngstown
  • slotface
  • Durango
  • kavyap
  • DreamBathrooms
  • JUstTest
  • normalnudes
  • osvaldo12
  • tester
  • GTA5RPClips
  • cubers
  • everett
  • tacticalgear
  • ethstaker
  • provamag3
  • anitta
  • Leos
  • cisconetworking
  • lostlight
  • All magazines
    •