ElixirConf 2022 - Wednesday Morning
Topics
11:10am to 11:55am
45 minutes
Keynote
José Valim
12:05pm to 12:30pm
25 minutes
Track 1
Title: Robust, Distributed, and Parallel Processing for Enormous Images Using SuperVisor, Node, Flow, Nx, and evision.
Speaker: Susumu Yamazaki
Topic: Data Processing
Audiences: Advanced
What do you use when you process enormous images? Of course, Python, Numpy, and OpenCV will be helpful for it, but don’t you want to speed it up by processing it in a distributed and parallel way? Elixir can do it: 1. You can replace Numpy and OpenCV with Nx and evision. 2. Node and Flow can make the processing distributed and parallel. 3. Supervisor makes it robust for crashing due to consuming much memory. This presentation will introduce satellite image processing for an information provision system of sediment disasters as an example case study shown at ElixirConf US 2020.
Track 2
Title: Porting Legacy Backend Services into Elixir, Seamlessly
Speaker: Yiming Chen
Topic: Legacy Systems
Audiences: Advanced | Intermediate
We as developers have all faced a legacy system, no one wants to or dares to change its code, yet it still somehow runs and supports an important piece in our business. What’s worse is when the original developers have all left the team and no one is familiar with the code and the language anymore. We had several services like this kind at Tubi. We ported 2 of them into Elixir in 2022, while keeping the user experience the same during the whole migration process. This talk is about all the lessons we learned along the way.
Track 3
Title: Building a HEEX component library
Speaker: Matthew Platts
Topic: Heex
Audiences: Advanced | Intermediate | Beginner
Components have long been standard in Javascript frameworks, however are relatively new in Phoenix. Now, with HEEX released, it’s time to start building component libraries to supercharge your web app development. Firstly, using my experience in building Petal Components (https://hex.pm/packages/petal_components), I’ll walk you through how to build your own HEEX component library for your next project. Secondly I’ll show how powerful components can be by using our Petal Components library to build a web page with a form.
12:35pm to 1:00pm
25 minutes
Track 1
Title: Livebook smart cells are amazing
Speaker: Stephen Ball
Topic: Livebook
Audiences: Intermediate | Beginner
In May 2022 Livebook 0.6 introduced smart cells. These are units of abstraction to wrap complex code in a simple user interface which means we can make a lot of amazing things happen behind the scenes while presenting users with a smart workflow. In this talk we’ll go from Livebook itself to smart cells that pull complex data from APIs to feed into other smart cells for analysis.
Track 2
Title: Classifying Bot IP Addresses in Phoenix
Speaker: Michael Lubas
Topic: Phoenix
Audiences: Advanced | Intermediate | Beginner
This talk will cover: 1. A brief description of how bots cause problems in Phoenix apps, showing why someone would want to classify requests as coming from a bot. 2. The design of a plug to compare each incoming request’s IP against a radix tree of public cloud provider IP ranges. 3. Why persistent_term is better than ETS for this task 4. How to use this plug to prevent bot attacks in your Phoenix app This talk is based on the blog post - https://paraxial.io/blog/cloud-ips
Track 3
Title: The Elixir Of Web Scraping
Speaker: Constantin Cheptea
Topic: Web Scraping
Audiences: Advanced | Intermediate | Beginner
This talk is about how we implemented a simple crawler for an MVP for one of our clients. I will talk about how we used a Stream to implement the crawler, the Eager vs Lazy topic, applying concurrency to improve performance, understanding the difference between concurrency and parallelism and how it helped us optimise resources. I will also allocate some time to cover ethical web crawling.
1:05pm to 1:30pm
25 minutes
Track 1
Title: Make a Soundcloud like app in a week and make it fit in a seven dollar box
Speaker: Miguel Michelson
Topic: Phoenix
Audiences: Intermediate | Beginner
This talk will be about the development of an open-source audio streaming platform (like Soundcloud). I will talk about why I’ve chosen Elixir and how I’ve put into the mix my previous work on some ported Rails framework gems (active_storage and active_job). This is the sequel of my talk given recently at ElixirconfEU 2022 titled “porting a Rails app to Elixir in two months,” and I will show some things regarding the LiveView uploads, streaming files from controllers to a cloud bucket with byte-range support, and some thoughts on the Elixir community here in Chile which is growing and helping to build this platform collectively.
Track 2
Title: Dx - a pragmatic inference system based on your Ecto schema
Speaker: Arno Dirlam
Topic: Ecto
Audiences: Advanced | Intermediate
Dx is an Elixir library that allows adding inferred properties to Ecto schemas, defined by rules based on fields, associations or other inferred properties. You can then load/query them as if they were Ecto fields or associations using Dx’s API. It allows you to write declarative and easy-to-read domain logic by defining WHAT the rules are without having to care about HOW to execute them. Under the hood, Dx’s evaluation engine loads associations as needed using dataloader and can even translate your logic to Ecto queries directly. Dx can be used for various use cases, from authorization to state machines, and excels in areas where a lot of (edge) cases need to be respected. Status: Dx has been used in production internally at Team Engine since April 2021, but is still in the early stages as a ready-to-use library. Repo: https://github.com/dx-beam/dx Docs: https://hexdocs.pm/dx/ Want to work on Dx with us? https://tinyurl.com/engine-infer-dev
Track 3
Title: Cryptography and Elixir
Speaker: Héctor Iván Patricio Moreno
Topic: Cryptography
Audiences: Advanced | Intermediate | Beginner
We all know that our digital security and anonymity relies on sophisticated algorithms that allows us to maintain secrecy and store information securely. We will talk about the main types of cryptographic algorithms (block and stream ciphers, hashes, MAC’s) and the right way to use each one. Finally we will talk about the cryptography libraries developed for Elixir and show examples about how to use it correctly.
1:30pm to 2:00pm
30 minutes
Schedule
- 11:10am to 11:55am - Keynote
-
12:05pm to 12:30pm
- [ ] Track 1 - 3 - Robust, Distributed, and Parallel Processing for Enormous Images Using SuperVisor, Node, Flow, Nx, and evision
- [ ] Track 2 - 5 - Porting Legacy Backend Services into Elixir, Seamlessly
- [x] Track 3 - 8 - Building a HEEX component library
-
12:35pm to 1:00pm
- [ ] Track 1 - 8 - Livebook smart cells are amazing
- [ ] Track 2 - 5 - Classifying Bot IP Addresses in Phoenix
- [x] Track 3 - 13 - The Elixir Of Web Scraping
-
1:05pm to 1:30pm
- [ ] Track 1 - 5 - Make a Soundcloud like app in a week and make it fit in a seven dollar box
- [x] Track 2 - 13 - Dx - a pragmatic inference system based on your Ecto schema
- [ ] Track 3 - 8 - Cryptography and Elixir
- 1:30pm to 2:00pm - Ask Us Anything with Test Double
Understanding these Notes
- Notes attempt to be in sequential order.
- Anything worth calling out falls to the bottom of the section, it may not belong in the stream of consciousness.
-
Using the block quote
>to show notes taken during rewatching the VODs. - I don’t want to modify the original notes much but this should allow me to somewhat annotate the previous sections.
- Allows us to also add new sections.
Notes
Keynote
There was a delay due to technical difficulties so some of us didn’t get the feed until probably 11:15/11:20. It felt like quite a few minutes.
> After watching the VOD, it ends up being close to the 10 minute mark. The first topic on the slide was PartisionSupervisor:after I believe. The very first topic from the keynote is dbg() and we got a history of how it dropped. That it knows the stack is huge and highly useful.
Bah I forgot to take notes as I can’t easily multitask how I want. This is a reminder to look at the VOD.
The dbg() Livebook stuff was insane, definitely had a “holy shit!” moment. I had to check myself from commenting that in the chat.
> I wasn’t aware of how the code of conduct handles something like that. It wouldn’t be harassment so it seems okay but it’s better to be professional in these settings.
> defstruct-ordered inspect is highly useful and one of my nitpicking pain points. Alphabetical order makes sense but I more often prefer the order of creation. I can’t really think of any reason I wouldn’t prefer this honestly.
> improve errors for binaries would be useful though I haven’t worked with binary information all that much. Knowing what is broken versus having a very educated guess is always a better experience.
> General improvements should be helpful considering how much I use Livebook. The error messages there are helpful but more is always better. Steps in slices being unified is big, I’m a huge proponent of improving the API consistently like this. Multi-line comments are small but necessary changes too. Everything called out here is worth highlighting but there’s definitely much more in the change log.
> I wanted to separately call out that I don’t understand why mix wouldn’t be available for releases. What is the use case for not including it by default. Could this be an opt in? How does Livebook include mix? Being in Laravel-land, artisan is always available after being deployed. I would have a much harder time doing absolutely everything on production machines without this in Laravel so I really do not understand this not being available. It would be helpful to uncover the reason, reinforce it in my brain, and look to see if we can work around this if possible.
> Three main areas of focus
-
Set theoretical types for Elixir
- Covered in ElixirConf EU Keynote, need to watch this
-
Developer and learning experience
- This is the main focus of this keynote
-
Machine learning
- Check out the talks from Sean Moriarity and Chris Grainger
Spawnfest 2022 is worth possibly entering, it’ll be happening in 43 days from now at https://spawnfest.org/
> Supabase and Fresha are hiring
Q & A
[@capitalist] QUESTION: Any thinking on how to link code on disk with a cell in livebook - or just copy/paste for now? Attached runtime is part of the puzzle, just need cell to fn mapping - sourceror splicing?
[@w0rddriven] @capitalist do you mean using Livebook as sort of an IDE for the code on disk, not the compiled code as a dependency?
[@josevalim] only copy and paste right now
[@capitalist]
@josevalim Thank you! @w0rddriven Or even editor tooling like hover a function to dbg in Livebook and you’re in the livebook, attached to runtime, code is there, changes can be synced back to file on disk. Would give really fluent back and forth ability - reminds me of the stuff smalltalkers do.
Brainstorm
What if this were a SmartCell, that behaved a lot like dbg()?
It would need to detect all of the code cells in a particular notebook. Using a select toggle, blocks 1, 4 and 7 for instance could be saved to {location}. Look into the behavior of Livebook’s own autosave notebook. How does it handle a previous file? Overwrite? It’s really important to have consistency.
Collapse doc comments? How would markdown between code work? Think of the inverse, how would I split up a module file? Where would the boundaries be? Could you infer a natural “shape”? A language like C# has collapsey bits but Elixir? I don’t believe so. If there’s 100 public functions, how do you quickly segregate? Split by line #?
This can easily get messy but it seems worth it for teaching a code base.
12:05pm to 12:30pm
- [ ] Track 1 - 3 - Robust, Distributed, and Parallel Processing for Enormous Images Using SuperVisor, Node, Flow, Nx, and evision
- [ ] Track 2 - 5 - Porting Legacy Backend Services into Elixir, Seamlessly
- [x] Track 3 - 8 - Building a HEEX component library
I was eating lunch with the family but I want to check out all of these. I’m a fan of the Heex component visualization icons, those are helpful. I’d likely want to segregate components how frameworks like Laravel do them.
There are clever techniques like supporting hooks in dead views that is worth a deeper dive.
> Track 1 Robust, Distributed, and Parallel Processing for Enormous Images Using SuperVisor, Node, Flow, Nx, and evision comes in right around 00:57:00 of the VOD
12:35pm to 1:00pm
- [ ] Track 1 - 8 - Livebook smart cells are amazing
- [ ] Track 2 - 5 - Classifying Bot IP Addresses in Phoenix
- [x] Track 3 - 13 - The Elixir Of Web Scraping
Missed a good bit of this due to lunch cleanup. This is a different approach than SpiderMan but it looks very close. This would be useful for my own techniques around Req usage.
Ethical web crawling
- Read the ToS and robots.txt
- Identify yourself when sending requests
- Set time-outs
- Don’t take advantage
- Contact site owners to inform them so they can opt out
> Track 1 Livebook smart cells are amazing comes in right around 01:26:00 of the VOD
This talk is highly useful to the idea of a smart cell synchronizing code cell contents to disk. Smart cells are a UI representation that converts 1-way to a code cell. What would the code cell for “synchronize all the contents above me” look like?
-
A smart cell is three concepts
- front end user experience
- Elixir code that translates the user experience to generated Elixir code
- Any underlying Elixir code supporting the generated Elixir code
This talk goes over the basis for adding behaviors and tackling each warning in sequential order. Such a perfect approach to how I normally would do something like this.
quote do then Kino.SmartCell.quoted_to_string() is the best approach for the to_source call, this will convert code to whatever format the smart cell expects.
Holy shit, at 01:44:46 it shows that for the JS he’s importing Vue 3 from vue@3.2.26/dist/vue.esm-browser.prod.js. Since Vue uses single file components and my current favorite frontend framework, this really speaks to my style.
Recap: What makes a smart cell?
-
A Javascript frontend
- Presents the user interface for attributes that direct the generation of Elixir code
- Communicates to/from the backend module
-
An Elixir module using some Kino behaviors
- Controls the lifecycle of the smart cell
- Communicates to/from the frontend
- Translates the attributes from the frontend into the source code of the Elixir code cell
-
Any Elixir code supporting the generated code
- Provided by the smart cell Elixir module directly
- Or by the smart cell package
- Or by dependencies of the smart cell package
Some cool smart cells xkcd ExDuck - Generates markdown from DuckDuckGo’s instant answer API. How it converts to markdown may be highly useful Tanium
1:05pm to 1:30pm
- [ ] Track 1 - 5 - Make a Soundcloud like app in a week and make it fit in a seven dollar box
- [x] Track 2 - 13 - Dx - a pragmatic inference system based on your Ecto schema
- [ ] Track 3 - 8 - Cryptography and Elixir
Started a little late. Already took screenshots of what he covers and all of the preloaded relationships. Knowing what someone would chain together makes it easy to break things out in similar categories.
I need to keep an understanding of declarative and imperative programming in my brain at all times to help me stick to one path, or show clear demarcation. I don’t know if I clearly know when I veer into either approach or if I prefer one over the other.
Dx looks really promising and it reminds me of Laravel/Eloquent models, though they follow more of a repository pattern. You can define model properties as columns, virtual fields as getter/setter methods, or methods like is_archived() that looks at internal state (though typically that would be a virtual is_archived bool in most cases). Phoenix generators tend to focus these sort of changes around the individual repo modules. I really need to look into this with Livebeats again and really compare Laravel to Phoenix to fully understand this.
> Track 1 Make a Soundcloud like app in a week and make it fit in a seven dollar box comes in right around 01:56:00 of the VOD
Rauversion will have a ton of useful examples I can use for my new music notification idea, and possibly others. ActiveStorage seems highly useful and it would be interesting to compare to Livebook’s storage encapsulation (I was about to say facade and it wouldn’t be a far off comparison to Laravel).
It’s difficult to take notes to this mostly because I just wanna see the techniques used and it’s more of a walk through of the code base. This makes for an interesting talk genre that I didn’t know I prefer until now.
The sticky live render option will likely be useful to keep components around during redirects. I often create UI “shells” this approach fits into.
1:30pm to 2:00pm - Ask Us Anything with Test Double
I didn’t find the link until I happened upon it in the Introductions section. A system wide notification like the late start would’ve been nice but it’s understandable. This was on Hopin last year and changing event tools will always have some growing pains.
There isn’t a lot of time between tracks for the random hiccup like this. The zoom link was https://us02web.zoom.us/j/87912011786 though it showed it was recorded. Hopefully it’ll be available soon.