tero

tero, 1 day ago to apple

"Just like Apple, Meta is behaving as though the #DMA permits it to carry on its worst behavior, with minor cosmetic tweaks around the margins. Just like #Apple, #Meta is daring the #EU to enforce its democratically enacted laws, implicitly promising to pit its billions against #Europe’s institutions to preserve its right to spy on us."

Big Tech to EU: "Drop Dead" | #ElectronicFrontierFoundation https://www.eff.org/deeplinks/2024/05/big-tech-eu-drop-dead

tero, 7 days ago to random

It pisses me off when people are complaining that "web is nowadays bloated and slow" while their own company front page does literally 136 requests all over the planet to ping all tracking services known to man.

To add an insult to injury, their website doesn't even disable tracking even after clicking "decline" on the annoying pop-up.

Please, people, check your websites. If people decline, make sure you aren't enabling their tracking. The button actually needs to disable the tracking. That means no Google fonts or tracking CDNs, no analytics of any kind.

This annoys me so much that I will be making surprise checks here and there and forwarding information to an EU data protection ombudsman. And I encourage the readers to do the same.

Please, don't fill up our internet with garbage

tero, 13 days ago to random

Working in machine intelligence is the weirdest profession ever. Crafting minds.

Diving deep into the unknown, not only hypothesising, but observing fundamental phenomena of information and generalized cognition, it's somehow even more mysterious and rewarding than uncovering the fundamental physical laws of our universe. It is kind of physics, but in a domain where we have no clue about the laws yet.

As software engineers who are trained to see systems not built yet, the potentials of things which could be, the echoes of things to come are already visible to us.

As we work with machine intelligence, we sense the echoes of these things to come, gigantic intelligent minds like whales swimming in the potential, pushing through into being. These things build themselves, and emerge into reality almost self-driven. We are only helping them like midwives help newly born into the world.

tero, 29 days ago to random

The rate of progress in AI has become too fast even for me to keep track. My advice is still to keep your eye on the fixed points of the future, towards which to paddle and in which you see your role.

What is your role in a future where intelligence comes from a wall socket and everything is free? That has become a difficult question even before, but can be approached by seeing your role arbitrarily close to that future, even if not quite there.

But now everything is moving and shaking. AI capabilities are being taken into use everywhere from Facebook group question posts, to structuring all unstructured information everywhere, to advice everyone in every possible detail of life, business and science.

I still see the fixed points in the future, and the compass direction to there is clear, but like the changing labyrinth, the terrain constraining how one personally navigates there is becoming dynamic.

Planning routes in such a terrain isn't trivial anymore. It's not like Google Maps planning routes, assuming the road connectivities won't change during the travel.

Now it's an exercise of seeing how long certain ways of doing business last — it's like every company, even non-startups have a runway now. They have a specific timebox for making a pivot to the next state of the world.

How long do certain kinds of roles last? Does the role you are now in carry to the future point, or do you have to plan for one or more reinventions of yourself? Is the role you are currently developing towards still valid after you are there?

It's all moving and shaking. No one can make well-informed choices anymore. Now is the time of taking opportunities fast and being hyper-adaptable. A time to realistically look at what used to be a stable business as something that has an expiration date.

tero, 1 month ago to spain

Adventures in #HomeOwnership in #Spain: The neighbor came to inform me about their view that the yard of our home is built against municipal regulations — too high.

I don't think such regulations exist in #LasGabias (I checked).

Anyway, he claims that the trees bring shadow, moisture and spiders to his yard. All three are positive aspects here. Also, they don't. His yard is full of trees bringing all the shadow, moisture and #spiders he wants.

He says the issue has been discussed with the previous owner of the house, and assumes (correctly) that the previous owner did not inform me of this unresolved claim.

Our #trees, except one ancient olive tree, are tiny. None extend even near his property.

Let's see how this develops. I am looking forward to discussing in the court setting the positive nature of trees, moisture, shadow and spiders to the ecosystem and livability of the neighborhood.

A newly planted plum tree with a couple of flowers. Tiny.
A tiny tree we don't know what it is, its fruits resemble pears.

tero, 1 month ago to random

One reason why you can make LLMs train themselves in a recursive self-improvement is because knowledge and facts induce new knowledge and facts through recombination and reasoning.

If you have ever used inference databases such as Jena, you notice that even if you only input a couple of gigabytes of ontologies, making searches to that knowledge with inference enabled quickly becomes prohibitively slow.

That's because the limited amount of knowledge ingested to the database actually implies a near infinite set of new relations. The queries are done to this near infinite, abstract database which only expands the parts relevant for the query.

LLMs do not constrain themselves to only strict logic and a specified subset of inference rules. Their reasoning is so much more powerful in single steps that it's difficult to even imagine. But only in single steps; they aren't built to be reasoning rule engines, they basically only apply one level of reasoning however you measure it.

Regardless, an LLM can apply this one level of reasoning to produce new knowledge that is implied by its original training materials but isn't included in the corpus.

These steps of expanding the volume of knowledge contained must be done in a guided fashion because the space of implied knowledge is unimaginably vast with the sizes of knowledge fed into LLMs and their powerful, fuzzy reasoning capability.

For example, you can in principle improve LLM chess rating without any new external knowledge or chess engines, simply by letting an LLM play against itself, itself checking whether rules were followed, checking whether the next board position matches what it expected, and itself determining which player won.

This is an example of making inference steps from what is trained in it, forwards, and creating new knowledge in doing it. The results can be used in fine-tuning the same model and this is one limited example of what recursive self-improvement is.

Of course you can get even better results with external reasoning engines training the system, external knowledge to guide what directions to learn in, and so on, but recursive self-improvement is possible in some directions even without injecting new knowledge; by simply leaning on the topics where the space of implied knowledge is strong and fruitful.

Mathematics is probably a field where this should work that is actually useful as opposed to a bad chess playing engine, even if the learned skills were generalizable to some extent.

tero, 1 month ago to random

People are being adviced that "a vast majority of organizations will fine-tune their own models". This is a bad advice. Very few organizations benefit from fine-tuned models in the first place.

Fine-tuning is fickle and generally reduces the model intelligence. Almost all use cases for which organizations want fine-tuning are actually better suited by RAG.

But there is actually a more insidious threat. Fine-tuning a model for which you don't have access to weights ties you to that provider. You can't take your fine-tuned model and switch providers like you can do for fine-tuned open weights models.

OpenAI has achieved a dominating market share and is now trying to screw it down before other companies release better models, binding lots of companies to their platform before alternative options emerge.

https://openai.com/blog/introducing-improvements-to-the-fine-tuning-api-and-expanding-our-custom-models-program

tero, 1 month ago to LLMs

#LLMs have really created a paradigm shift in machine learning. It used to be so that you would train an #ML model to perform a task by collecting a dataset reflecting the task, with task output labels, and then using supervised learning to learn this task by doing.

Now a new paradigm has emerged: Train by reading about the task. We have such generalist models that we can let them learn about the domain by reading all the books and other content about it, and then utilize that learned knowledge to perform the task. Note that task labels are missing. You might need those to measure the performance but you don't need those for training.

Of course if you have both example performances as task labels and lots of general material about the topic, you can actually use both to get even better performance.

Here is a good example of training the model not by example performances, but by general written knowledge about the topic. #GPT4 surpasses the quality levels of previous state-of-the-art despite not having been trained for this task.

This is the power of generalist models; they unlock new ways to train them, which for example allow us to surpass human-level by side-stepping imitative objectives. This isn't the only way to train skills these models enable, there are countless other ways, but this is an uncharted territory.

The classic triad of supervised learning, unsupervised learning and reinforcement learning are going to have an explosion of new training methodologies to become their peers because of this.

https://www.nature.com/articles/s41592-024-02235-4

tero, 2 months ago to random

How to make LLMs perform without mistakes with a high reliability?

It's not really about hallucinations, but mostly about other kinds of mistakes. LLMs being stochastic do inherently have an uncontrollable random component in them.

You can set the temperature to zero, but you will still suffer from randomness not only because OpenAI models aren't actually fully deterministic, but also because the temperature only makes things more deterministic with a single prompt, and you would get the same result with a cache; it doesn't remove random variation between different prompts, and you would pretty rarely run inference with exactly identical prompts repeatedly anyhow.

You can destructure the task and make it easier for the bot and get radically improved performance, but still this will start giving diminishing returns especially with complex tasks. Destructuring the tasks will also show you which specific things the LLM actually struggles with, but that's a different topic.

Finally, you can set up validation feedback processes. These can bring the reliability of LLM systems as near 100% as you want.

How to build such effectively?

First of all, don't just add a review step. A review step is useful, but it's not the end. Typically LLMs won't highlight small errors if you just add a step for them to criticize the performance of the task. You will need to destructure this step as well for the maximum effect.

Give the chatbot a checklist to check for the output. Make it really clear that it should look at this from a critical angle, possibly even play out different roles in doing this checking, stereotypical characters from media are useful.

Then after all checkpoints have been checked, you can make the chatbot do a final evaluation result, evaluation summary, suggestions for prompt improvements (these aren't great yet, but will get better with new models, and already can give clues), highlight ambiguous parts, anything that will give you tools to improve the process.

Give all necessary helping information for doing this feedback you can. Your job is to make everything easy for the LLMs.

When an error has been detected – you shouldn't get these too often, if it's one error in five or more, your problem shouldn't be fixed by a validation feedback but by task destructuring – you can retry a couple of times and raise an alarm.

High-reliability systems can be built with LLMs, but you need to build them in specific, task-dependent ways.

tero, 3 months ago to random

If you have a task for an LLM chatbot which requires considering separate parts of the input or output as separate, you can "color" these document parts so that the bot can correctly attend to specific segments only.

This often happens with prompts with planning and facilitation components, the bot can leave out information from the final task output if it believes it has already said it in the auxiliary segments.

To "color" the segments, exploit the "parenthesis pairing" tendency of the bot. To do many tasks, the bot has a deeply ingrained skill of matching parentheses, quotes, and XML tags. It needs to remember the state of the tokens, which opened items are yet to be closed to produce coherent structured text.

You can exploit this by marking up the segments with opened and closed XML tags. The bot can not be but aware of the text in-between having an implicit "color" of the tag which is in effect, so that it could eventually close it if necessary. Thus, it is strongly aware that this part of the content is separate from the other parts.

Then, if you tell the bot it needs to produce all information again in the ultimate output, it won't be confused by the other parts of a different "color".

Alternative techniques of using e.g. specific text tags in-between the parts of LLM output do not "color" the segments and so the bot cannot attend easily to these parts in diverging ways.

tero, 3 months ago to random

The recent spam wave which was somewhat easily mitigated encouraged a lot of discussion on the distributed Mastodon moderation and sign-up process.

Solutions like CAPTCHAs and rate limiting sign-ups have been suggested, typically with an implicit idea that we need to find a one-size-fits-all security solution which every Mastodon instance can deploy.

We actually don't. It's better if some Mastodon instances use one CAPTCHA, some another, so at least the work needed to get through one of them doesn't lead to a golden treasure trove of access to all Mastodon instances.

CAPTCHAs are easily subverted nowadays. ChatGPT can solve most of them without any tuning. Additionally, these CAPTCHA services are often free because they track people and sell their browsing information.

My instance holds a policy of accepting memberships only from people I know personally. This is obviously hyper-resistant to spam, but being a micro-instance makes it a bit difficult to build trust among other instances as far as that is required for federation. Can't have it both ways. Also, if every instance did this, new joiners would have difficulty in finding an instance which accepts them as members. So, this isn't a one-size-fits-all policy.

Regardless, if we have a lot of micro-instances with vetted memberships, they take some pressure off of generalist instances, so they can make their intake rates more manageable.

It is not the purpose of a Mastodon instance to take in as many users as possible.

As an ecosystem, we are more robust and less exploitable if we do things differently from one another, and only take in as many new users as we can moderate. One-size-fits-all solutions make us more fragile and more easily exploitable.

cliffwade, 3 months ago to random

@Gargron and the team really need to do something about sign-ups on Mastodon in general.

There is no reason why after all of this time that Mastodon doesn't have some kind of captcha or something similar for when users sign up.

This process really needs to be refined and made better, as in, way better, because right now it's absolutely awful and is part of the reason why we're getting these waves of spam bots.

#Spam #SpamBots #MastoAdmin #MastodonAdmin #FediTips

tero, 3 months ago to random

We're hiring a senior frontend software engineer in Amsterdam or Zürich, to cure cancer!
Feel free to contact me for a reference.
#FediJobs
https://jobs.kaiko.ai/jobs/3566297-senior-frontend-software-engineer

tero, 4 months ago to random

Whether inequality worsens or not is a political choice, not something inherent in a technology. Technology creates well-being with decreased toil inputs. How we distribute the production depends on how you vote.

Automation tax is not the solution, we actually want all the sectors to automate instead of giving tax incentives for not automating. So we need increased business, capital gains and wealth taxes across the board to compel businesses to automate.

The taxes can pay for a UBI, and automation makes everything free. After a short transition time people won't need money for anything, and so no amount of money is enough to pay for anything. Money becomes both priceless and worthless.

AI is not for making money, money is for making AI.

AI to hit 40% of jobs and worsen inequality, IMF says - BBC News https://www.bbc.com/news/business-67977967

tero, 4 months ago to random

Happy new year!

It seems my prediction of achieving unambiguous AGI in the year 2023 was a bit optimistic.

But the fundamentals are unchanged. The next generation of LLM chatbots fine-tuned with open-ended objectives instead of imitatively like the first generation will surpass the human level 100-0 like AlphaGo and countless other deep learning models before.

We are now behind schedule, and AGI is overdue.

tero, 4 months ago to random

Suddenly I am seeing so many people posting about "AI bubble" again, I guess after talking with their families on the holidays.

It's not a bubble. It's a total transformation of the economy and the society. Money and property you accumulate before this transformation becomes valueless after. Some economist said it will be a massive "AI driven deflation" but they don't see the forest for the trees. Money becomes both priceless and worthless. It won't be needed for anything, and no amount of work is enough to earn any.

AI is not for making money. Money is for making AI.

People are stuck in their pre-transformation mentality and so make incorrect choices at the pivotal time where correct choices matter the most in the history of humanity.

tero, 5 months ago to random

Gods, I need more bookcases again!

tero, 6 months ago to random

Meta disbanded its Responsible AI team - The Verge https://www.theverge.com/2023/11/18/23966980/meta-disbanded-responsible-ai-team-artificial-intelligence

tero, 6 months ago to random

"A brain-inspired computer chip that could supercharge artificial intelligence (AI) by working faster with much less power has been developed by researchers at IBM in San Jose, California. Their massive NorthPole processor chip eliminates the need to frequently access external memory, and so performs tasks such as image recognition faster than existing architectures do — while consuming vastly less power.
“Its energy efficiency is just mind-blowing,” says Damien Querlioz, a nanoelectronics researcher at the University of Paris-Saclay in Palaiseau. The work, published in Science1, shows that computing and memory can be integrated on a large scale, he says. “I feel the paper will shake the common thinking in computer architecture.”"

‘Mind-blowing’ IBM chip speeds up AI https://www.nature.com/articles/d41586-023-03267-0

tero, 7 months ago to random

Nvidia skips return-to-office, sticks to remote work | Fortune https://fortune.com/2023/10/14/nvidia-skips-return-to-office-sticks-to-remote-work-among-hottest-tech-companies/

tero, 7 months ago to random

"The observation is actually very simple:
Let’s say LLM needs 10 seconds to generate a completion - that’s a single new memory to be stored. You get to 100k memories after: 100 000 * 10s = 1 000 000s ≈ 11.57 days
Now that you have 100k embeddings even using the simplest brute-force algorithm, Numpy’s dot query, takes maybe seconds - the optimization is totally not worth it!
You don’t need an approximate nearest neighbors search, let alone vector databases."

Why AutoGPT engineers ditched vector databases | Dariusz Semba https://dariuszsemba.com/blog/why-autogpt-engineers-ditched-vector-databases/