SebastianM6L,
@SebastianM6L@mastodon.social avatar

what is the preferred way of creating data ingestion pipelines in ?

I have a few systems where I need to invest the data from an curious what ways are preferred or considered as good vs mine 🤣

ghickman,
@ghickman@mastodon.social avatar

@SebastianM6L my preference is to build it up in layers:

  • optional raw data validator (depends on trust in the source)
  • parser to convert raw data to internal representation (I love attrs for the storage here)
  • serialisation layer
  • entry point layer
  • management command for local testing/cron
  • optional webhook handler

Depending on your situation these layers can be anything from modules down to single lines!

SebastianM6L,
@SebastianM6L@mastodon.social avatar

@ghickman Thanks for the feedback.
I think I took a similar approach.
At least not to far off.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • django
  • DreamBathrooms
  • magazineikmin
  • tacticalgear
  • thenastyranch
  • ethstaker
  • rosin
  • love
  • Youngstown
  • slotface
  • ngwrru68w68
  • kavyap
  • osvaldo12
  • mdbf
  • everett
  • megavids
  • InstantRegret
  • cubers
  • modclub
  • tester
  • khanakhh
  • GTA5RPClips
  • cisconetworking
  • Durango
  • normalnudes
  • provamag3
  • anitta
  • Leos
  • JUstTest
  • All magazines