2019-06-03 11:01:50	dkliban	It's 11:01 ... please say !here if you plan to participate in the meeting ... even if it's just following along
2019-06-03 11:01:56	dkliban	!here
2019-06-03 11:01:58	jsherrill	!here
2019-06-03 11:02:02	ttereshc	!here
2019-06-03 11:02:27	ipanova	!here
2019-06-03 11:02:32	dkliban	let's give 2 more mins for anyone else that wants to join
2019-06-03 11:04:03	x9c4	!here
2019-06-03 11:04:12	quba42	!here
2019-06-03 11:04:31	bmbouter	!here
2019-06-03 11:04:36	ggainey	!here
2019-06-03 11:04:49	ttereshc	quite a quorum
2019-06-03 11:04:54	dkliban	this is great
2019-06-03 11:05:32	bmbouter	ja
2019-06-03 11:05:38	dkliban	We can begin. I was hoping we could spend this hour talking about how the user is going to express what needs to be migrate - Migration Plan
2019-06-03 11:05:48	dkliban	but i see that we have a lot of content in the Questions section
2019-06-03 11:06:02	ggainey	heh
2019-06-03 11:06:18	dkliban	So let's begin by going through them one by one  
2019-06-03 11:06:30	dkliban	i am hoping that with the content in the etherpad we already have them answered
2019-06-03 11:06:41	dkliban	from line 20:     Are exporter distributors needed in Pulp 3? Are they needed to be migrated?
2019-06-03 11:06:54	dkliban	jsherrill: do you have any thoughts on that?
2019-06-03 11:07:15	jsherrill	dkliban: we do not need migration for those
2019-06-03 11:07:24	dkliban	jsherrill: wonderful news!
2019-06-03 11:07:42	dkliban	does anyone else think that we need to migrate export distributors?
2019-06-03 11:08:05	ttereshc	community might want to
2019-06-03 11:08:17	ttereshc	I don't know how broadly it's used
2019-06-03 11:08:38	dkliban	i don't think we should say that we will never migrate export distributors
2019-06-03 11:08:49	dkliban	but we can de-prioritize them for now
2019-06-03 11:08:53	ttereshc	maybe we can throw a note on a pulp-list and see if there is a demand to migrate it
2019-06-03 11:09:02	ipanova	what about rsync distributor- into which category does it fall?
2019-06-03 11:09:18	dkliban	it is it's own category
2019-06-03 11:09:21	bmbouter	that falls into a pulp3 "exporter" I think
2019-06-03 11:09:32	dkliban	yes, it would be an exporter in Pulp 3
2019-06-03 11:09:33	bmbouter	exporters in pulp3 ship data to remote destinations
2019-06-03 11:09:47	ipanova	ok
2019-06-03 11:10:23	ipanova	ttereshc: +1 to a note on pulp-list
2019-06-03 11:10:39	dkliban	ttereshc: would you be interested in sending out a note like that?
2019-06-03 11:10:40	bmbouter	+1
2019-06-03 11:11:01	ttereshc	sure and it seems like bmbouter thinks the same :)
2019-06-03 11:11:04	dawalker	!here
2019-06-03 11:11:24	mikedep333	!here
2019-06-03 11:11:28	dkliban	ttereshc: great! i'm putting you down for that AI
2019-06-03 11:11:38	dkliban	let's go on to the next question 
2019-06-03 11:11:54	dkliban	How exactly will pulp3 migrate the content units via the api (or does it)? -- line 26
2019-06-03 11:12:25	dkliban	The current plan is to run the migration 'tasks' on Pulp 3
2019-06-03 11:13:05	jsherrill	is that a 'task' ?
2019-06-03 11:13:08	dkliban	and not use any REST APIs. just direct access to Pulp 2 DB and Pulp 3 DB
2019-06-03 11:13:18	bmbouter	yeah it already is guaranteed a pulp3 connection, it's got the python3 runtime we want, and it guarantees you won't have two migrations runnig concurrently
2019-06-03 11:13:48	bmbouter	jsherrill: it would be a task that is dispatched by katello in your case
2019-06-03 11:15:16	dkliban	jsherrill: does that answer your question?
2019-06-03 11:15:20	jsherrill	yes
2019-06-03 11:15:38	dkliban	Next question on line 29:     Does the pulp3 server need to talk to the pulp2 api? how will auth work?
2019-06-03 11:16:34	dkliban	there are answers in the etherpad. but the short version is No. shared filesystem and direct access to the DB will allow Pulp 3 to have all data it needs from Pulp 2
2019-06-03 11:17:13	dkliban	Next question is on line 34:     How will we find out what got migrated where?
2019-06-03 11:17:41	dkliban	Pulp 3 is going to create tables in the database that provide mappins between Pulp 2 and Pulp 3 resources
2019-06-03 11:18:30	jsherrill	dkliban: is that data going to be exposed over pulp3's rest api?
2019-06-03 11:18:39	dkliban	jsherrill: with regard to querrying, Pulp 3 could provide REST API or just direct DB access
2019-06-03 11:18:51	dkliban	that is up for discussion
2019-06-03 11:19:00	jsherrill	-1 to direct db access ;)
2019-06-03 11:19:03	ttereshc	would it be a part of migration tool?
2019-06-03 11:19:13	dkliban	I can definitely see the migration tool adding extra REST APIs to pulpcore
2019-06-03 11:19:21	jsherrill	it honestly doesn't sound like the migration tool is going to do very much
2019-06-03 11:19:23	ggainey	dkliban: we'll also want to have a way to decide "i'm done with these tables" and purge the db (something a little less frightneing to users than 'drop table *;' :) )
2019-06-03 11:19:56	dkliban	jsherrill: what do you mean?
2019-06-03 11:20:42	jsherrill	dkliban: we can keep going, it just sounds like the pulp3 api and tasking system is doing 99% of the work here
2019-06-03 11:21:20	dkliban	jsherrill: let's keep going and come back to that in a little bit
2019-06-03 11:21:24	jsherrill	k
2019-06-03 11:21:28	bmbouter	the "migration tool" is all the code that would run in there
2019-06-03 11:21:45	dkliban	and it comes as separate packages
2019-06-03 11:21:53	dkliban	that you can remove after you are done 
2019-06-03 11:21:54	bmbouter	so we can move into what it does and spend more time on "how it runs" later
2019-06-03 11:22:02	jsherrill	maybe these terms need to be better defined :)
2019-06-03 11:22:07	dkliban	line 40: [quba42] If pulp 2 and pulp 3 are not feature equivalent (in the DB) then how can a direct migration possibly work?
2019-06-03 11:22:32	dkliban	quba42: do you want to elaborate?
2019-06-03 11:22:41	dkliban	what do you mean by equivalency here?
2019-06-03 11:23:04	quba42	If pulp 2 db does not have the information I want to have in pulp 3?
2019-06-03 11:23:20	dkliban	quba42: then this tool can't help you
2019-06-03 11:23:32	dkliban	quba42: do you have a specific example?
2019-06-03 11:24:23	quba42	Surely, the migration tool needs to create a state in pulp3 as if the things that are added had been added by pulp 3 but using only pulp 2 information.
2019-06-03 11:25:10	dkliban	one thing i can think of is Artifacts and their checksums
2019-06-03 11:25:28	dkliban	Right now Pulp 2 does not have that information and Pulp 3 will need to calculate all the checksums
2019-06-03 11:25:57	dkliban	and this information will be generated by Pulp 3 during the migration
2019-06-03 11:26:26	dkliban	Importers in Pulp 2 will be created as Remotes in pulp 3
2019-06-03 11:26:35	bmbouter	yeah can we write those mappings
2019-06-03 11:27:02	dkliban	yes, we can on the etherpad
2019-06-03 11:27:16	bmbouter	also there are phases kind of planned, and it may be good to structure our convo in the same way
2019-06-03 11:27:24	bmbouter	so phase 1 is really about artifacts and migrating those
2019-06-03 11:28:23	dkliban	quba42: has your question been answered? 
2019-06-03 11:28:57	quba42	More or less.
2019-06-03 11:29:23	quba42	I don't have a concrete example right now, but I am pretty sure we will find something in pulp_deb...
2019-06-03 11:29:31	dkliban	good .. we are approaching 30 after the hour and i was hoping we could spend some time talking about the Migration Plan 
2019-06-03 11:30:02	dkliban	there are more questions and i am hoping we can answer them as we discuss the Migration Plan
2019-06-03 11:30:29	dkliban	I put some notes about the migration plan starting on line 63
2019-06-03 11:30:29	ttereshc	+1
2019-06-03 11:31:10	dkliban	The Migration Plan will be a JSON structure that a user can submit into Pulp 3 to begin the migration process
2019-06-03 11:31:37	dkliban	The user is going to use the Migration Plan to express the state of Pulp 3 using references to Pulp 2
2019-06-03 11:31:51	ggainey	dkliban: is the MP something the user creates entirely? or do we envision a task that will spit out a 'just take the defaults' proposal, that the user can vet and then submit?
2019-06-03 11:32:29	dkliban	ggainey: Pulp 2 will provide an API that generates this default migration plan.
2019-06-03 11:32:49	ggainey	awesome, thanks
2019-06-03 11:32:50	dkliban	this default MP will migrate everything that's possible to migrate
2019-06-03 11:32:50	bmbouter	I imagine it will have a repos section, and a content section
2019-06-03 11:33:05	ttereshc	submit into Pulp3?  can you elaborate
2019-06-03 11:33:18	quba42	Does Katello provide it's own migration plan, for translating Content Views into repo versions?
2019-06-03 11:33:21	ttereshc	did you mean that migration toll will work with that MP
2019-06-03 11:33:29	ggainey	bmbouter: users?
2019-06-03 11:33:43	jsherrill	quba42: yes, katello would generate one itself
2019-06-03 11:34:22	dkliban	ttereshc: yes, as metioned earlier, Pulp 3's tasking system will be used to perform the migration. so the user will need to submit the MP to a REST API that is provided by the migration tool package
2019-06-03 11:34:57	ttereshc	ggainey, I guess not until we support multiple users in  Pulp3
2019-06-03 11:35:15	ggainey	dkliban: so the workflow is, "Add produce_migration_plan API to pulp2, call pmp API, edit results to taste, submit pmp.json to pulp3, wait for tasks to finish, evaluate results", yeah?
2019-06-03 11:35:19	ggainey	ttereshc: ah, kk
2019-06-03 11:35:28	x9c4	will there be a dry-run?
2019-06-03 11:35:43	dkliban	x9c4: there should probably be such an option
2019-06-03 11:35:53	dkliban	x9c4: what information do you expect it to produce?
2019-06-03 11:36:04	ggainey	yeah, at a minimum some kind of sanity-check for the migration-plan
2019-06-03 11:36:22	ttereshc	what should it do? what can be checked?
2019-06-03 11:36:42	x9c4	Well syntax for sure.
2019-06-03 11:37:01	ggainey	well, 'valid json according to schema', to start :) - and then 'do the specified entities actually exist in the pulp2 instance you want me to start from'? 
2019-06-03 11:37:49	ggainey	also, what about capacity? can we even figure out something like "you want me to migrate X TB of data from your pulp2 instance, into a pulp3 that only has 500Mb of storage available"?
2019-06-03 11:38:02	x9c4	Maybe 'Do any of the to be generated entities collide in some way?'
2019-06-03 11:38:07	ggainey	trying to think of the kinds of things that have bitten me in past lives
2019-06-03 11:38:17	ggainey	oo, yeah that's a good one
2019-06-03 11:38:36	ggainey	"Do you really mean to migrate all of your repositories into the same repo in pulp3?"
2019-06-03 11:38:43	ggainey	the heuristics could get exciting
2019-06-03 11:39:07	dkliban	x9c4: yes, we will have a dry-run option that will do some sanity checking
2019-06-03 11:39:25	dkliban	x9c4: i think we can do some planning for that in a separate session
2019-06-03 11:40:41	dkliban	I was thinking the MP could be broken up into 5 sections: Content, Repositories, Repository Versions, Remotes, and Distributions
2019-06-03 11:41:07	dkliban	The sections that interest me the most are teh Repositories and Repository Versions
2019-06-03 11:41:09	jsherrill	dkliban: publications ?
2019-06-03 11:41:25	dkliban	jsherrill: yes, publications
2019-06-03 11:41:25	bmbouter	this sounds good
2019-06-03 11:41:28	x9c4	This is 6 ;) But i would add Artifacts.
2019-06-03 11:41:39	ttereshc	Repository Versions section will be created by user only, right?
2019-06-03 11:41:53	dkliban	ttereshc: what do you eman?
2019-06-03 11:42:08	ttereshc	dkliban, can you generate the MP with that section?
2019-06-03 11:42:20	ttereshc	based on data in pulp2
2019-06-03 11:42:51	dkliban	ttereshc: the default MP would have a Repository Version section that says 'create a repository version for each repository based on it's current content set in pulp 2'
2019-06-03 11:43:05	jsherrill	so one version for each repository
2019-06-03 11:43:07	jsherrill	by default
2019-06-03 11:43:09	jsherrill	that makes sense
2019-06-03 11:43:12	ttereshc	ok, I got it
2019-06-03 11:43:28	ttereshc	I thought it would be empty and not explicit
2019-06-03 11:43:29	ttereshc	thanks
2019-06-03 11:43:29	dkliban	yes, but i suspect katello needs to be able to express more
2019-06-03 11:43:34	jsherrill	for sure
2019-06-03 11:43:39	ttereshc	yeah yeah
2019-06-03 11:43:42	ttereshc	I understand
2019-06-03 11:43:50	dkliban	jsherrill: please share what you would to describe
2019-06-03 11:44:17	dkliban	you = katello ... s/would/would want/
2019-06-03 11:44:42	quba42	Katello wants to change copies of repositories in pulp 2 into versions of pulp 3 (and then have a publication/distribution for each) 
2019-06-03 11:44:48	jsherrill	dkliban: we'd describe, for some Repository X in pulp3, we want a list of version from repositories [a,b,c,d,e,f]
2019-06-03 11:45:09	dkliban	that sounds very reasonable
2019-06-03 11:45:19	jsherrill	quba42: yep exactly, for all of those we'd want publications and distributions defined
2019-06-03 11:45:30	bmbouter	this sounds good
2019-06-03 11:46:02	x9c4	Additional distributions for the lifecycle environments, i guess
2019-06-03 11:47:17	jsherrill	x9c4: good point, we may want to add more distributions onto it, but i guess we could do that post migration ourselves
2019-06-03 11:47:21	dkliban	jsherrill: are you planning to treat each lifecycle env as a separate repository?>
2019-06-03 11:47:41	jsherrill	dkliban: no, actually with pulp3, a repository version is just exposed with a new distribution 
2019-06-03 11:47:52	jsherrill	to make it available in a lifecycle env
2019-06-03 11:48:19	jsherrill	so a migrated repository version may need more than one distribution
2019-06-03 11:48:41	dkliban	so we need to be able to express that relationship in the MP
2019-06-03 11:49:23	ttereshc	does pulp has all info needed to create a distribution for a lifecycle env?
2019-06-03 11:49:42	jsherrill	if we specified what distributors in pulp3 to create it from yes
2019-06-03 11:49:45	dkliban	there is a distributor in pulp 2 for it
2019-06-03 11:50:18	dkliban	jsherrill: since it's a different repository in pulp 2, there is a separate distributor for it
2019-06-03 11:50:33	dkliban	for it = for a lifecycle env
2019-06-03 11:50:35	jsherrill	right, let me type out an example structure
2019-06-03 11:51:38	bmbouter	yeah
2019-06-03 11:51:45	bmbouter	I think we need some example structures
2019-06-03 11:51:51	jsherrill	line 85
2019-06-03 11:52:01	dkliban	i seee it
2019-06-03 11:54:23	dkliban	that makes sense t ome jsherrill 
2019-06-03 11:54:33	ttereshc	jsherrill, should line 85 be a list as well?
2019-06-03 11:54:51	jsherrill	you could only specify one pulp2 repo per version
2019-06-03 11:54:56	jsherrill	to pull the unit list and publication
2019-06-03 11:55:21	ttereshc	and how do you express content views?
2019-06-03 11:56:03	x9c4	a content view contains a list of repo versions.
2019-06-03 11:56:17	jsherrill	that's how we would.  for each version of the content view, for each repo, we'd create a pulp3 repository version, each with 0..N distributions
2019-06-03 11:57:00	x9c4	right, content view version... i think it lives only in katello.
2019-06-03 11:57:06	jsherrill	yep
2019-06-03 11:57:33	ttereshc	I meant, where do you describe this: "some Repository X in pulp3, we want a list of version from repositories [a,b,c,d,e,f]"?
2019-06-03 11:57:54	jsherrill	oh, the structure i write i figured would be part of a larger strucutre
2019-06-03 11:57:57	jsherrill	let me expand it a bit
2019-06-03 11:58:19	ttereshc	thank you
2019-06-03 11:58:54	x9c4	Is it possible, to mix content from different plugins in one Repository?
2019-06-03 11:59:12	dkliban	we have not thought about that
2019-06-03 11:59:19	dkliban	i would probably think no
2019-06-03 11:59:24	dkliban	is there a use case for that?
2019-06-03 11:59:41	dkliban	we are also at the end of our hour
2019-06-03 11:59:52	x9c4	Just curious. It would be _like_ a content view based in pulp.
2019-06-03 12:00:18	ipanova	dkliban: at least we do not prohibit this in pulp3 
2019-06-03 12:00:42	dkliban	that's right. we do not prohibit it  
2019-06-03 12:01:02	ipanova	dkliban: so depending on the json data it might happen we'd have a mixture in a repo
2019-06-03 12:01:25	dkliban	i would like to bring this meeting an end  ... i will send out meeting minutes later today
2019-06-03 12:01:55	dkliban	jsherrill: thank you for providing the descriptions
2019-06-03 12:02:31	ggainey	dkliban: thanks for kicking this off
2019-06-03 12:02:50	dkliban	i will send out an email when i think we are ready to discuss more
2019-06-03 12:02:59	dkliban	i am hoping that will be soon
2019-06-03 12:03:07	bmbouter	great
2019-06-03 12:03:13	ttereshc	+1
2019-06-03 12:03:18	ipanova	dkliban: thank you
2019-06-03 12:03:26	ttereshc	thanks for facilitating it
2019-06-03 12:03:39	bmbouter	yes ty
2019-06-03 12:04:30	x9c4	Thanks to the moderator