This post is mostly intented to remind myself in the future, what fosdem 2016 looked like for me, but maybe somebody else find it useful as well.
In
the first keynote, about systemd, Lennart was speaking mostly about
stuff that will happen in 2016 in systemd and particularly dnssec. He
began by introducing recent move of the project to github and introduction of CI.
Systemd is default in almost all distros, with no many rants around..
"Are we boring now?", he asked.
Then he went through networkd, mentioning it's much more automatic and smaller than NetworkManager, nspawn to be different from Docker because nspawn is not only for microservices. sd-dhcp now used also by NetworkManager, although it's stil not public nor a supported component yet.
Unified control group hierarchy fixes issues that the old nonunified implementation had, but the new one is not used much yet, because API has changed and it will break stuff.
systemd and Where We Want to Take the Basic Linux Userspace in 2016
Then he went through networkd, mentioning it's much more automatic and smaller than NetworkManager, nspawn to be different from Docker because nspawn is not only for microservices. sd-dhcp now used also by NetworkManager, although it's stil not public nor a supported component yet.
Unified control group hierarchy fixes issues that the old nonunified implementation had, but the new one is not used much yet, because API has changed and it will break stuff.
DNS resolving, previously done by glibc, is going to
be centralized now in systemd-resolved. Caching will make it better but big thing is to
have dnssec ready.
The dnssec in systemd-resolved checks just integrity, no confidentiality. Chain
of trust coming from TLD zones bellow should eventually check services like ssh, tls, pgp, certificates. However, only 2% of most
popular websites use dnssec, eventhough main TLDs were signed in2010 already.
Interestingly, 8.8.8.8, a popular Google's DNS server does.
Validation on client is not done though, because it's slower and clients do not support it. It is important for ipv6 though.
Private networks are problem because those cannot be validated from the top level, since the sites do not exist officialy. Another problem is that ISP's DNS servers and end-user's routers are crap.
Systemd's approach is to give up when dnssec cannot be verified, which opens vulnerability, yes. But when it is successful, clients are reported and pgp, tls, ssh or others may be happy, so full dnssec enabled internet is not a blocker.
Question in the end asked about Lennart's previous plans to change the way how applications are distributed and whether this is still the plan. Answer was that most of needed technology is done, but distributions have not adopted it yet.
Colin Charles from MariaDB talked about packaging. He gave us an overview of whole MySQL ecosystem, mentioning that MySQL has been always more open-source product, not a project, which I agree.
He mentioned the FOSS exception that MySQL may be distributed under GPL and proprietary license at the same time.
Validation on client is not done though, because it's slower and clients do not support it. It is important for ipv6 though.
Private networks are problem because those cannot be validated from the top level, since the sites do not exist officialy. Another problem is that ISP's DNS servers and end-user's routers are crap.
Systemd's approach is to give up when dnssec cannot be verified, which opens vulnerability, yes. But when it is successful, clients are reported and pgp, tls, ssh or others may be happy, so full dnssec enabled internet is not a blocker.
Question in the end asked about Lennart's previous plans to change the way how applications are distributed and whether this is still the plan. Answer was that most of needed technology is done, but distributions have not adopted it yet.
Distributions from the view of a package
He mentioned the FOSS exception that MySQL may be distributed under GPL and proprietary license at the same time.
Fedora was often mentioned as example of
distributions that care about packages quality, mentioning some
particular mailing list threads, bugs and we several times saw Tom Lane's name there, since he used to maintain MySQL for long time.
Software Collections also mentioned as solution for differences between support period of distributions and MySQL.
Some statistics showed that default version in long-term support distributions is important (RHEL and MariaDB).
Docker and juju shows stats, fedora and OpenSUSE used to do that as well but do not anymore, although it would be good for upstream to prioritize their resources.
Some platform issues were mentioned, like bundling vs. not including some extensions.
He mentioned that they care about being informed about distribution bugs in MariaDB upstream, even be on CC sometimes. He mentioned how many downstream fixes are in fedora and elsewhere, and that this needs to be fixed.
Software Collections also mentioned as solution for differences between support period of distributions and MySQL.
Some statistics showed that default version in long-term support distributions is important (RHEL and MariaDB).
Docker and juju shows stats, fedora and OpenSUSE used to do that as well but do not anymore, although it would be good for upstream to prioritize their resources.
Some platform issues were mentioned, like bundling vs. not including some extensions.
He mentioned that they care about being informed about distribution bugs in MariaDB upstream, even be on CC sometimes. He mentioned how many downstream fixes are in fedora and elsewhere, and that this needs to be fixed.
MySQL Group Replication or how good theory gets into better practice
Tiago Jorge
talked about project still in MySQL labs (labs.mysql.com), Group Replication is supposed to remove fuzz from users when dealing with
replication fail-over. Group communication primitives concept from early
90s was inspiration for group replication.. The process includes a
certification procedure, which is about asking other nodes whether they
do not work with the data we want to work with by any chance.
Group replication needs some special options, like binlog, GTID, algorithm for something and we need a replication user. Set up requires UUID for specification (because this will be used by GTID instead of server ID) of groupname and specification of some nodes addresses, not necessarily all of them. Performance schema can say how many and which members are there, and it also includes various stats, like transactions stats, ... rejoining a member can use sync method that only applies what is not yet done. Alos, catch-up process does not block donor.
Main features: Cloud friendly, integrated, automated, no big overhead, ..
More info in mysqlhighavailability.com, like this one: http://mysqlhighavailability.com/mysql-group-replication-hello-world/
No support for online ddl though.
Question: Does it support thousand of nodes? Answer: There is a linear degradation because of communication limits.
Btw. It is type of eventual consistency.
Group replication needs some special options, like binlog, GTID, algorithm for something and we need a replication user. Set up requires UUID for specification (because this will be used by GTID instead of server ID) of groupname and specification of some nodes addresses, not necessarily all of them. Performance schema can say how many and which members are there, and it also includes various stats, like transactions stats, ... rejoining a member can use sync method that only applies what is not yet done. Alos, catch-up process does not block donor.
Main features: Cloud friendly, integrated, automated, no big overhead, ..
More info in mysqlhighavailability.com, like this one: http://mysqlhighavailability.com/mysql-group-replication-hello-world/
No support for online ddl though.
Question: Does it support thousand of nodes? Answer: There is a linear degradation because of communication limits.
Btw. It is type of eventual consistency.
ANALYZE for statements: MariaDB's new tool for diagnosing the optimizer
Sergei
Petrunia talked about explain and optimize statements. Query plan
sometimes does not correspond with reality and we need to show stats,
like row stats, especially stats per table. MySQL's ANALYSE is inspired by EXPLAIN ANALYZE from PostgreSQL and Oracle, and in comparison to EXPLAIN alone, it
not only optimizes the statement, but also executes it and shows some
stats. It shows comparison between rows guessed and real rows count
read. A general use case is that big discrepancies are worth
investigating.
Filtered fields can tell how many rows were read and then discarded, which suggests to add indexes, which is always a trade-off between faster reads or faster writes.
ANALYZE statement can fix table's histogram that positively influences execution plan.
EXPLAIN may return json which might be read better, values prefixed with r_ are from ANALYZE part. We can see information like buffer sizes used or cache hit ratio as well.
An example showed by Sergei explained how to use that all to see which sub-queries caused the biggest performance issues.
ANALYZE works fine to show also range selects.
Current cons are that explain statement sometimes lies. ANALYZE may be much more right.
During Q&A, it was mentioned that histograms are better for selectivity than cost model, which is what MySQL uses them for.
Rene
Cannao spoke about experiences with GTID in Dropbox. Typically we have
at least 2 slaves, Dropbox has two masters (one basically a slave of the
original) and every master also has own slave. Binary logs are
important, classic approach uses file and position.
Filtered fields can tell how many rows were read and then discarded, which suggests to add indexes, which is always a trade-off between faster reads or faster writes.
ANALYZE statement can fix table's histogram that positively influences execution plan.
EXPLAIN may return json which might be read better, values prefixed with r_ are from ANALYZE part. We can see information like buffer sizes used or cache hit ratio as well.
An example showed by Sergei explained how to use that all to see which sub-queries caused the biggest performance issues.
ANALYZE works fine to show also range selects.
Current cons are that explain statement sometimes lies. ANALYZE may be much more right.
During Q&A, it was mentioned that histograms are better for selectivity than cost model, which is what MySQL uses them for.
Rolling out Global Transaction IDs at Dropbox
In case of non-GTID
deployment, on slaves, the file names and positions are not same as on
the master, which may be tricky especially when we have nested
replication, because the second slave does not know where to start
replication after the middle slave crashes. GTID consists of UUID of
source and transaction ID and since it is the same across the all
cluster, it makes the recovery easier -- the slaves simply know where to
begin the recovery. Slave just uses master_auto_position =1.
The use case Rene showed, utilized also enforce_gtid_consistency, log_slave..., gtid, binlog... options.
He went through procedure of enabling GTID either offline (requires to turn off the cluster for some time) or online (without restarting all servers at once).
Sometimes there might be problem with zoombie slaves, time-outing.
Serge
Frezefond talking about Connect plugin in MariaDB, that can access data
from other sources. Features like ability to specify options when
creating a table of type connect or auto discover structure of the file
makes this plugin quite easily usable. With table type ODBC, we can use
syntax like PRIOR statement that does not exists in MySQL. It also
allows to create a triger that runs some command on an entry execution.
We can also query different databases at the same time, like Oracle DB and PostgreSQL. The XML option supports xpath syntax to describtion of which tag corresponds with which column.
JSON does not replace integrated JSON support (already in MySQL, comming soon in MariaDB) but can add extrrnal JSON source into the db.. The structure of JSON needs to be addressed, and it is done by setting a starting point for the table. Jpath spec is used for aggregation..
Most of the stuff are writable also, but the performance is not always perfect.
Raghavendra
Prabhu from Yelp was talking about orchestrating. K8s has some requirements, like server-client design,
cattle, not pet approach, horizontal scaling rather that vertical,
statelessness in databases and elastic scalability, like scaling both
directions. We should also switch to declarative rather than imperative
thinking.
There are a lot of container technologies available. Galera shortly introduced, that it uses optimistic approach when solving concurency conflicts and supports automatic node sync. CAP theorem for galera is true for CP.
Stateless DBs, problem for big data.
Kubernetes shortly introduced.
PaaSTA supports a lot of existing tools and technologies.
Galera node is equivalent with pod.
Examples of k8s configuration files were shown.
A
physical board game created by Chris Ward should be available freely in
next months. CMS from start, doc in Markdown, web made by Jeckyl. Also
pdf generation should be done at once with web, pandoc used for it.
Latex also used to design cards, someone said it is like css for
printing.
So far no big graphic, ImageMagic used for something.
Question asked, whether it should be a template for others. Answer was yes, eventhough there might be work done to make this ready..
Giuseppe
Maxia talking about docker and MySQL. Services on shared resources is
not good. We pay for VMs. Container is visualization system but is no
VM. Several things to keep in mind when working with containers carrying
databases. Never modify running container and deploy ready made
containers only. We can change options by passing them to cmd.. another
way is use config server by bind mounting a .cnf file.
bit.ly/my-rep-samples.
Question about secret passwd, Guisepe suggests passwd from file.
Sveta
Smirnova talked about features in MySQL users asked for and how oracle
is well responsive to user's requests. For example after last years'
Markus Winand's talk, who mentioned that MySQL does not include FILTER
statement. She was able too add it using rewrite query engine. Then she
took a bit closer look at how this was implemented, really showing C
code. It showed how to work with MySQL's memory management. The filter
query is rewritten by regular expression to CASE WHEN syntax. The new
rewritten query is stored and some flag must be set.
Another example showed how comments in query might be rewritten to three queries -- setting a variable from comment and resetting it after the query is executed. This included usage of THD structure which allows to change variables quite easily.
Then, she showed how to add some new variables and she just mentioned it is possible to do much more like locks, or implement even httpd server inside mysql, etc..
Langdon
talked about Software Collections in world of unikernels and
containers. Answer was simply yes. Problem is breaking apps to
microservices. You might want multiple versions of same components
inside container as well. SCL bring you simpler packaging, than if
getting components by hand. Good for migration or for logical separation
of similar components.
Q: was it developed for OpenShift? Not only.
Q: SCLs in fedora? Yes, it makes sense, there are same reasons as in RHEL, but from opposite point of view, we need older versions or versions that are in RHEL.
Q: Why not use just dockerfiles? With rpm as middle step it is easier to verify, that packages work as expected.
Ludovic
Courten talking about GNU Guix, because keeping SW environment polished
is not easy. Distro is stateful system. Is it hopeless? Someone puts
whole distro into application container as well. Using layers in docker
is like using frozen pizza for cooking own flavored pizza.
Functional package management might be answer. We have complete image of how packages are composed. Resulting path where padkage is placed into, is hash of all deps.
We can resolve all deps exactly.
We can install things without root.
Intallation and removal of packages may be done in one transaction with Guix.
Every transaction creates a generation, like a named state where we can go back to.
Additional package manager, but this one is cool.
Gimp deps is a big mess, not easy. Search path is huge, like really huge.
There is also guixos, similar to nixos, that boots quite nicely from generated file.
Recipy is similar to RPM spec but is functional, basic skeleton might be stolen from Nix, but there are differences.
The use case Rene showed, utilized also enforce_gtid_consistency, log_slave..., gtid, binlog... options.
He went through procedure of enabling GTID either offline (requires to turn off the cluster for some time) or online (without restarting all servers at once).
Sometimes there might be problem with zoombie slaves, time-outing.
MariaDB CONNECT Storage Engine: Simplify heterogeneous data access
We can also query different databases at the same time, like Oracle DB and PostgreSQL. The XML option supports xpath syntax to describtion of which tag corresponds with which column.
JSON does not replace integrated JSON support (already in MySQL, comming soon in MariaDB) but can add extrrnal JSON source into the db.. The structure of JSON needs to be addressed, and it is done by setting a starting point for the table. Jpath spec is used for aggregation..
Most of the stuff are writable also, but the performance is not always perfect.
Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes
There are a lot of container technologies available. Galera shortly introduced, that it uses optimistic approach when solving concurency conflicts and supports automatic node sync. CAP theorem for galera is true for CP.
Stateless DBs, problem for big data.
Kubernetes shortly introduced.
PaaSTA supports a lot of existing tools and technologies.
Galera node is equivalent with pod.
Examples of k8s configuration files were shown.
Programming a Board Game
So far no big graphic, ImageMagic used for something.
Question asked, whether it should be a template for others. Answer was yes, eventhough there might be work done to make this ready..
MySQL operations in Docker: A quick guide for the uninitiated
Question about secret passwd, Guisepe suggests passwd from file.
Introducing new SQL syntax and improving performance with preparse Query Rewrite Plugins
Another example showed how comments in query might be rewritten to three queries -- setting a variable from comment and resetting it after the query is executed. This included usage of THD structure which allows to change variables quite easily.
Then, she showed how to add some new variables and she just mentioned it is possible to do much more like locks, or implement even httpd server inside mysql, etc..
Do Software Collections still matter?: With containers, unikernels, and all the new hotness?
Q: was it developed for OpenShift? Not only.
Q: SCLs in fedora? Yes, it makes sense, there are same reasons as in RHEL, but from opposite point of view, we need older versions or versions that are in RHEL.
Q: Why not use just dockerfiles? With rpm as middle step it is easier to verify, that packages work as expected.
Reproducible and Customizable Deployments with GNU Guix: Why "app bundles" get it wrong
Functional package management might be answer. We have complete image of how packages are composed. Resulting path where padkage is placed into, is hash of all deps.
We can resolve all deps exactly.
We can install things without root.
Intallation and removal of packages may be done in one transaction with Guix.
Every transaction creates a generation, like a named state where we can go back to.
Additional package manager, but this one is cool.
Gimp deps is a big mess, not easy. Search path is huge, like really huge.
There is also guixos, similar to nixos, that boots quite nicely from generated file.
Recipy is similar to RPM spec but is functional, basic skeleton might be stolen from Nix, but there are differences.
Sunday?
There are no other notes from Sunday, because my plane was leaving too early, but still this year's Fosdem was great, year to year better and better. Can't wait for next year already.