2019-06-13 10:00:32 ttereshc !here 2019-06-13 10:00:34 ggainey !here 2019-06-13 10:00:57 ggainey oh wait - no pulpbot here? 2019-06-13 10:01:04 dkliban not yet 2019-06-13 10:01:06 dkliban !here 2019-06-13 10:01:08 ggainey kk 2019-06-13 10:03:39 dkliban i guess it's just the 3 of us 2019-06-13 10:03:51 dkliban let's wait another 2 mins 2019-06-13 10:04:03 ttereshc hmm..let's wait a bit 2019-06-13 10:04:05 ttereshc yeah 2019-06-13 10:04:18 ttereshc bmbouter confirmed that he'll join 2019-06-13 10:04:49 bmbouter here 2019-06-13 10:05:10 ttereshc \o/ 2019-06-13 10:05:37 * ggainey cheers wildly 2019-06-13 10:05:50 dkliban wooot 2019-06-13 10:05:54 dkliban alright ... let us begin 2019-06-13 10:06:18 dkliban ttereshc: has put together some use cases that start on line 184 2019-06-13 10:07:37 dkliban is there any particular one of those that anyone wants to discuss? 2019-06-13 10:08:39 dkliban i was hoping to discuss "As a user, I can migrate distributors and publications" on line 229 2019-06-13 10:08:46 dkliban but i think we should have jsherril here for that 2019-06-13 10:09:07 dkliban we need to determine what information will be migrated for protected repositories 2019-06-13 10:09:26 dkliban and what content guards will be created 2019-06-13 10:10:34 ttereshc agreed 2019-06-13 10:10:53 ttereshc I have a question about content vs no content migration 2019-06-13 10:10:53 --> quba42 (~qubap42@213.148.108.114) has joined #pulp-2to3-sig 2019-06-13 10:11:03 dkliban sure 2019-06-13 10:11:05 ttereshc we haven't discussed it in the past 2019-06-13 10:11:14 dkliban what's your queston? 2019-06-13 10:11:17 ttereshc does it make sense for everyone? 2019-06-13 10:11:28 ttereshc do we need to provide this option? 2019-06-13 10:11:28 dkliban the concept in general? 2019-06-13 10:11:55 dkliban I think users will want to have the option of just migrating their repository definiteions and the importers needed to sync 2019-06-13 10:12:09 dkliban and then having pulp just sync the content 2019-06-13 10:12:12 ggainey so it made sense to me when I was thinking that 'migrating content' meant copying/moving/duplicating all the actual content-blobs 2019-06-13 10:12:12 --> x9c4 (~0x9c4h@nd114.net.atix.de) has joined #pulp-2to3-sig 2019-06-13 10:12:12 ttereshc fwiw, it's on line 222 As a user, I can migrate repositories and their importer configuration to sync later (no content option) 2019-06-13 10:12:56 dkliban i think it's a useful option 2019-06-13 10:13:01 ttereshc so how do we migrate distributors/publications in this case? Do we migrate them anyway? 2019-06-13 10:13:16 dkliban not publications, just distributors 2019-06-13 10:13:28 ttereshc to have a relative path set? 2019-06-13 10:13:31 dkliban yes 2019-06-13 10:13:41 ttereshc and potentially certguards 2019-06-13 10:13:44 dkliban yes 2019-06-13 10:14:13 ggainey can we explicitly define what is and is not possible if you don't migrate content? 2019-06-13 10:14:19 ggainey in the doc 2019-06-13 10:14:39 dkliban yes, we can add language about that 2019-06-13 10:15:05 ttereshc I just added on line 238 that we migrate distributors only 2019-06-13 10:15:35 dkliban ttereshc: i think this will be a challenging user experience though 2019-06-13 10:15:37 ggainey what does 236 mean? "add available (in Pulp 3) content" ? 2019-06-13 10:15:41 x9c4 !here 2019-06-13 10:15:47 dkliban welcome x9c4 2019-06-13 10:15:54 * ggainey waves 2019-06-13 10:16:00 ttereshc ggainey, that's my next question/topic 2019-06-13 10:16:12 ggainey heh 2019-06-13 10:16:15 ttereshc dkliban, what do you mean? where is it challenging? 2019-06-13 10:16:17 dkliban ttereshc: the reason i think so is because once the repos, remotes, and distributors are migrated it will be hard for the user to tell what belongs together 2019-06-13 10:16:33 ttereshc right 2019-06-13 10:16:35 dkliban and i know that we will provide APIs for looking up mappings to pulp 2 2019-06-13 10:17:08 dkliban but i still think the user will find him/herself in a situation where they have to figure out what repository goes with which remote and which distribution 2019-06-13 10:17:21 dkliban and that is just how pulp 3 works 2019-06-13 10:17:48 ttereshc yeah if not for migration, then for subsequent syncs/publishes 2019-06-13 10:17:58 ggainey sounds like a requirement on 'output of migration process' 2019-06-13 10:18:23 ttereshc ggainey, +1 mappings between pulp2 and pulp3 are the must 2019-06-13 10:18:29 dkliban yes 2019-06-13 10:18:31 ggainey yah 2019-06-13 10:18:38 bmbouter this all sounds good 2019-06-13 10:19:03 dkliban but i still think we should support this use case 2019-06-13 10:19:43 dkliban let's go to the next topic ttereshc 2019-06-13 10:19:43 ggainey from the end-user POV, 'how do i know what to expect', 'how do i know how the migration went', and 'how do i know where my old-familiar-entities ended up' are really important for this kind of migration 2019-06-13 10:19:53 * ggainey gets off of soapbox 2019-06-13 10:19:55 dkliban lol 2019-06-13 10:20:31 dkliban ttereshc: did you want to discuss how content in pulp 3 gets associated with repository versions? 2019-06-13 10:20:31 ggainey hrm, epad keeps disconnecting - anyone else having a prob, or is just my network? 2019-06-13 10:20:41 dkliban it's the pad 2019-06-13 10:20:46 ggainey ah kk thanks 2019-06-13 10:20:57 ttereshc ggainey asked what "add available (in Pulp 3) content" is 2019-06-13 10:21:17 ggainey yus 2019-06-13 10:21:25 dkliban yeah .... ggainey, if the user has already run the migration task before there may be some content in pulp 3 already 2019-06-13 10:21:30 ttereshc and I put it there to discuss whether we want to do it or not and whther it makes sense 2019-06-13 10:21:31 ggainey ahhhhhhhhhhhh, ok 2019-06-13 10:21:47 ttereshc yes what dkliban said 2019-06-13 10:21:49 dkliban so if the second time the user is running the task it's possible that the user asks not to migrate content 2019-06-13 10:21:56 dkliban but there is already content there 2019-06-13 10:22:01 ggainey dkliban: well - that opens a can of worms 2019-06-13 10:22:16 ttereshc which one? there are many :) 2019-06-13 10:22:44 ggainey if I start a migration with one set of parameters/filters/decisions, how much sense does it make to 'pick it up in the middle' after changing your mind about what to do? 2019-06-13 10:22:55 dkliban we initially stated that the migration plan with declerative 2019-06-13 10:23:02 dkliban and i think we should really continue with that in mind 2019-06-13 10:23:14 ttereshc +1 2019-06-13 10:23:28 dkliban so that means that if the second time you run the migration task you say don't want content migrated, we should remove content that was already migrated 2019-06-13 10:23:34 ggainey i'm ok with that, as long as we spend a lot of time investigating/cleaning-up the P3 state every time a migration starts 2019-06-13 10:23:54 bmbouter +1 to declarative 2019-06-13 10:24:00 dkliban ggainey: yes, i think we have to start each migration task with cleanup 2019-06-13 10:24:18 dkliban and just general checking of the state of things 2019-06-13 10:24:21 ggainey it's more than just content, it's All The Things - which is ok, it's just going to require us to pay a lot of attention to edge-cases and interrelationships 2019-06-13 10:24:39 dkliban ggainey: we'll fix those in production! 2019-06-13 10:24:40 bmbouter I imagined much of it would include checks at various levels and migrate as needed 2019-06-13 10:24:46 ggainey this is all just ('just') implications on test-plans and QE 2019-06-13 10:24:51 ggainey kk 2019-06-13 10:25:05 ttereshc dkliban, can you elaborate more on cleanup task 2019-06-13 10:25:28 ttereshc we discussed that if you ran a migration for the second time and something changed that new repo versions are created 2019-06-13 10:25:35 ttereshc and nothing is cleaned up 2019-06-13 10:26:13 ttereshc or do you mean if the MP changed, we clean up? 2019-06-13 10:26:23 dkliban ttereshc: no, if something changed and the MP says that there should be only 1 repository version for the repo, the existing version is deleted and a new one is created 2019-06-13 10:26:39 ggainey oh ugh, I hadn't even thought about the "pulp-2 state has changed since last we saw it" set of edge cases 2019-06-13 10:26:40 dkliban ttereshc: yes, if the MP plan changed we do cleanup to align things with the new MP 2019-06-13 10:27:02 ttereshc dkliban, please define cleanup 2019-06-13 10:27:46 ggainey dkliban: ttereshc : so Step 0 in migration is validate-MP, step-1 is se if MP changed, step-1.a is "if MP has changed, remove from P3 anything done last time that is no longer in MP" ? 2019-06-13 10:27:56 dkliban ttereshc: in case of migrating content and not: if the first time you said you want to migrate content the task is going to migrate content. if you run the migration task again and the MP now says that it doesn't want to migrate content, the content in pulp 3 is removed 2019-06-13 10:28:08 ggainey Step-2 is 'see if pulp-2 has changed', Step-3 is 'do the migration'? 2019-06-13 10:28:37 ttereshc dkliban, what is removed? all the content and artifacts? or just repo version? 2019-06-13 10:28:50 bmbouter I think of the machinery as having levels (like those levels ttereshc stated ^) 2019-06-13 10:29:05 ttereshc we should also be able to do it for a specific plugin 2019-06-13 10:29:10 bmbouter agreed 2019-06-13 10:29:14 dkliban ttereshc: if contnet is being removed, then artifacts, and content is being removed 2019-06-13 10:29:34 dkliban and repo versions 2019-06-13 10:29:44 bmbouter as it 'checks' the existing repo versions it can know if it needs to take action 2019-06-13 10:29:55 ggainey yeah we need to make sure implications are stated explicitly 2019-06-13 10:30:05 bmbouter and similarly, if it then finds an artifact/content is missing, it can migrate that 2019-06-13 10:30:13 bmbouter I'm picthing a just-in-time approach 2019-06-13 10:30:49 ttereshc we don't have typed repos, so how do we clean up all docker stuff without touching file repos? by analyzing the content? 2019-06-13 10:30:49 bmbouter and also the concept that if designed that way we can achieve incredible parallelization because the code can be run as easily linearly as it can in parallel 2019-06-13 10:30:55 ggainey bmbouter: disagree, since as a user I am a big fan of being able to have the tool tell me everything it's planning on doing before I pull the actual trigger 2019-06-13 10:31:04 dkliban ttereshc: they are typed in pulp 2. 2019-06-13 10:31:10 bmbouter ggainey: I think we would run it with dry run mode 2019-06-13 10:31:11 dkliban ttereshc: we have a notes field on repos in pulp 2 2019-06-13 10:31:54 ttereshc dkliban, yes, however we run cleanup task on pulp3, how do we identify what to clean up? 2019-06-13 10:31:55 ggainey bmbouter: ok - I heard 'just in time' and my brain said 'nondeterministic', but it occurs to me that dry-run just does All The THings without actually writing to disk/db - nevermind :) 2019-06-13 10:32:15 ttereshc dkliban, by the mapping for the pprevious migration plan? 2019-06-13 10:32:19 dkliban yeah 2019-06-13 10:32:26 ttereshc oh 2019-06-13 10:32:37 bmbouter ggainey: well the other issue is that any actual action plan would probably not be what happens a few moments later since the pulp2 system is always changing 2019-06-13 10:32:50 dkliban that's the biggest challenge ^ 2019-06-13 10:32:55 bmbouter "justin-time" 2019-06-13 10:33:09 ggainey at some point, as a user, you have declare it 'done', somehow 2019-06-13 10:33:22 bmbouter ggainey: you turn off your pulp2 system 2019-06-13 10:33:37 bmbouter and the MP runs one last time 2019-06-13 10:33:39 ggainey so as an admin, at some point after doing the bulk of the work in parallel, i'm going to put my p2 system in readonly and finish the job 2019-06-13 10:33:41 ggainey yeah 2019-06-13 10:33:47 bmbouter w/ the pulp2 database still on 2019-06-13 10:33:54 bmbouter ya! 2019-06-13 10:34:12 bmbouter and it's that final runtime which is the actual pulp2 -> pulp3 downtime (I think) 2019-06-13 10:34:46 ggainey can p2 be in 'readonly' and still work? is that a thing at all? 2019-06-13 10:34:50 dkliban yes 2019-06-13 10:34:58 dkliban it can serve content, but not generate anything new 2019-06-13 10:35:28 dkliban pulp 2 has workers that are always used to generate new repo/content 2019-06-13 10:35:30 ggainey ok - so the last step is 'p2 in readonly, migrate last time, validate p3 results', and then the only 'downtime' becomes the act of switching actual urls? 2019-06-13 10:35:35 ggainey yeah 2019-06-13 10:36:00 dkliban ggainey: yes, but there is down time for somoene wanting to use rest api 2019-06-13 10:36:11 ggainey sure 2019-06-13 10:36:13 bmbouter I figured step-wise 'migrate last time' == 'validate p3 results' 2019-06-13 10:36:27 dkliban yes, hte migration task should be doing the validation 2019-06-13 10:36:43 ttereshc dkliban, I'm still concerned about relying on the mapping from the last MP. Imagine Katello migrates file plugin and 1/2 year later docker plugin, we can't be sure that mappings for file are still preserved 2019-06-13 10:36:44 ggainey bmbouter: i guess I separate migration and validation - migration is what the task knows, validation is what the admin does to make sure our code hasn't f'd them up :) 2019-06-13 10:37:09 bmbouter but in terms of the user having confidence that the MP is going to achieve their business needs w/ continuity (e.g. tooling updates to use P3)... 2019-06-13 10:37:17 bmbouter it would be pretty sweet actually 2019-06-13 10:37:21 dkliban ttereshc: i see what you are saying 2019-06-13 10:37:37 dkliban ttereshc: i suspect that the second time there will not be any ISO (File) contnet in pul p2 2019-06-13 10:37:56 ggainey ugh 2019-06-13 10:38:01 bmbouter if they wen't live w/ the file plugin they can't re-migrate it 2019-06-13 10:38:09 bmbouter at least w/ the same MP 2019-06-13 10:38:13 ttereshc I agree 2019-06-13 10:38:16 ggainey can/should we put limits on MP-freshness? 2019-06-13 10:38:19 bmbouter the tool is giving them what they asked for in that case 2019-06-13 10:38:21 dkliban bmbouter: the concern is that if we do cleanup based on the MP, we will delete live content 2019-06-13 10:38:27 ggainey yeah 2019-06-13 10:38:51 dkliban so the MP must have 'file' plugin listed 2019-06-13 10:39:00 dkliban so that it's content doens't get deleted 2019-06-13 10:39:11 bmbouter dkliban: yes that is how I imagined it 2019-06-13 10:39:27 dkliban ttereshc is concerned that people will forget to do that 2019-06-13 10:39:29 ttereshc too dangerous imo 2019-06-13 10:39:38 ttereshc dkliban knows ttereshc very well 2019-06-13 10:39:55 dkliban i guess this is where the dry run option will tell them that their shit is about to get wiped 2019-06-13 10:40:10 ggainey hm, I see - what if we had a "additive only" switch on the MP? "Migrate the stuff in this MP< don't worry about anything in P3 that is not listed here"? 2019-06-13 10:40:48 dkliban then it's no longer declarative 2019-06-13 10:40:57 dkliban and a bit more complex 2019-06-13 10:41:03 bmbouter yeah and that is a key aspect to making the design workable 2019-06-13 10:41:24 ggainey hm 2019-06-13 10:42:29 ggainey the combination of multiple-runs, over an arbitrary lenght of time, that can delete from P3, suggests to me that the task needs to at a minimum generate A LOT of warnings and "are you SURE about this?" opportunities for the user 2019-06-13 10:42:43 bmbouter the issue is that the user has no way to give feedback also 2019-06-13 10:42:49 bmbouter it's running on a backend tasking system 2019-06-13 10:42:58 ggainey or we're just asking for angry users - yes, it's documented, yes, dry-run said it was gon' do it, but at the end of the day, their data is gone 2019-06-13 10:42:58 dkliban bmbouter: but the user can do a dry run 2019-06-13 10:43:05 ttereshc dkliban, dry-run in combination with "make backup of pulp3 db" might work 2019-06-13 10:43:11 ggainey yeah 2019-06-13 10:43:15 bmbouter yeah 2019-06-13 10:43:19 dkliban yes, definitely need to make backups 2019-06-13 10:43:20 bmbouter I hear the concern for sure 2019-06-13 10:43:33 bmbouter I just don't know what to tell a user who points a tool like this at a live production system and uses it wrong 2019-06-13 10:43:49 bmbouter and what I could do to help them in that situation without making many other situations more difficult 2019-06-13 10:43:51 ggainey bmbouter: but that describes about 50% of the likely users :) 2019-06-13 10:44:07 dkliban they will have to resync all their content 2019-06-13 10:44:13 bmbouter if I made knives I wouldn't make them duller because people can cut themselves worse 2019-06-13 10:44:23 ttereshc bmbouter, do you expect a migration tool to remove stuff from the new system? 2019-06-13 10:44:27 dkliban i hear that's actually more dangerous bmbouter 2019-06-13 10:44:32 ttereshc any theoretical migration tool 2019-06-13 10:44:41 ggainey bmbouter: except we work in an industry that tells ppl you don't need to know which is the sharp end to use our shiny new knives :) 2019-06-13 10:44:56 bmbouter me make the blade and handle the same color :) 2019-06-13 10:45:01 ggainey heheh 2019-06-13 10:45:23 bmbouter ttereshc: I think for it to be declarative it would remove items from P3 2019-06-13 10:45:32 ggainey dumb question - can the P3 side of things be "in active use" before you're done turning of your P2 system? 2019-06-13 10:45:34 bmbouter but I imagined it to be strongly plugin-by-plugin 2019-06-13 10:46:02 bmbouter as in the MP says maybe right at the top, hey, we're only dealing w/ this plugin type, and maybe these content types within that plugin 2019-06-13 10:46:09 dkliban ggainey: it can be ... but at some point you will switch the reverse proxy to point at p3 2019-06-13 10:46:23 bmbouter so any P3 content of a type not in the MP would be left alone 2019-06-13 10:46:39 ggainey dkliban: if the P3 instance is active, then deleting content the migration doesn't recognize becomes *much* scarier 2019-06-13 10:46:45 bmbouter so yes and no, it would remove content if it's in the MP, otherwise no 2019-06-13 10:47:07 bmbouter it's typed though. oh I migrated file 6 months ok, now let's migrate docker 2019-06-13 10:47:18 ggainey ah ok 2019-06-13 10:47:27 ttereshc bmbouter, If I understood you correctly, +1 to allow to migrate one plugin only 2019-06-13 10:47:36 bmbouter yeah I imagine this very plugin-by-plugin 2019-06-13 10:48:04 dkliban i am a bit lost ... 2019-06-13 10:48:09 ggainey I was hearing "if I find a plugin not-in-MP in P3, I'm'a delete it" - which sounds like not what you have in mind 2019-06-13 10:48:28 dkliban that's what i thought that plan was ^ 2019-06-13 10:48:29 bmbouter say you have 3 plugins 2019-06-13 10:49:08 bmbouter the MP you get from pulp2 shows the 3 plugins 2019-06-13 10:49:43 bmbouter but you just want to migrate 1, so you edit the MP to just include that 1 type of content, and the repos that deal w/ that type of content, and run it 2019-06-13 10:50:02 bmbouter then you put pulp3 into produciton w/ that content type and delete the stuff from pulp2 (manually) 2019-06-13 10:50:08 dkliban yep 2019-06-13 10:50:11 ggainey ok, so the MP isn't as comprehensively-declarative as I was thinking 2019-06-13 10:50:24 dkliban bmbouter: so 6 months later .. 2019-06-13 10:50:26 bmbouter then 6 months later you ask Pulp2 for a new MP and you do it again, just 1 content type 2019-06-13 10:50:30 ggainey (which is fine, just different than my understanding) 2019-06-13 10:50:47 bmbouter the content type and repos that were migrated to P3 with MP1 are left unchanged 2019-06-13 10:50:57 bmbouter but oh we screwed MP2 up 2019-06-13 10:51:00 bmbouter let's adjust and rerun it 2019-06-13 10:51:25 bmbouter it just checks and updates the stuff specified in MP2 2019-06-13 10:51:42 bmbouter I can see some issues w/ ^ plan technically... 2019-06-13 10:51:53 ggainey bmbouter: given that - do we want the MP structure to change somewhat, to make 'plugin' be a top-level entity? 2019-06-13 10:51:59 bmbouter but I'm focusing on the user experience and creating a safe one that allows for migration of this content now and that content later 2019-06-13 10:52:03 bmbouter that is what I wanted yes 2019-06-13 10:52:12 bmbouter if that is what we want 2019-06-13 10:52:27 ggainey bmbouter: that would seem to match this expectation better 2019-06-13 10:52:39 ggainey where the individual-plugin-stanzas would be declarative 2019-06-13 10:52:41 dkliban we definitely want to make it safer to migrate one plugin at a time 2019-06-13 10:52:47 ggainey yes please 2019-06-13 10:53:16 bmbouter so orphan cleanup in P3 must be content type filterable (feature wise) 2019-06-13 10:53:30 bmbouter the tool can never call generic orphan orphan cleanup if ^ plan is to work 2019-06-13 10:53:42 ttereshc great, I can update the doc, I'll ping in a min so you can verify if I captured it correctly 2019-06-13 10:53:49 bmbouter sweeeeet 2019-06-13 10:53:52 ggainey cool 2019-06-13 10:54:09 ggainey this is a great discussion, for me at least 2019-06-13 10:54:55 dkliban ditto 2019-06-13 10:55:58 ttereshc line 194 2019-06-13 10:55:59 dkliban i see the changes on line 195 2019-06-13 10:56:15 dkliban ttereshc: i like that better 2019-06-13 10:56:26 ggainey ttereshc: would plugins be a list of dictionaries? 2019-06-13 10:56:37 ttereshc oh right 2019-06-13 10:56:47 ggainey ok great 2019-06-13 10:56:53 ttereshc fixed 2019-06-13 10:56:54 dkliban yep 2019-06-13 10:57:17 dkliban ttereshc: that looks right to me ... and the migration task only cares about the plugins listed 2019-06-13 10:57:25 ttereshc yes 2019-06-13 10:57:29 dkliban it does not do anything to the content or repos of the plugins not listed 2019-06-13 10:57:35 ttereshc much safer for users 2019-06-13 10:57:36 bmbouter yes 2019-06-13 10:57:47 ttereshc and very explicit 2019-06-13 10:57:47 ggainey dkliban: aye - compare to list in prev-run, cleanup any overlap (only), then execute new-MP, yeah? 2019-06-13 10:57:51 bmbouter I like this much better 2019-06-13 10:57:56 ggainey concur 2019-06-13 10:57:57 dkliban mee too 2019-06-13 10:58:04 dkliban we are at the end of our hour 2019-06-13 10:58:16 ggainey feels like progress, thanks folks 2019-06-13 10:58:22 dkliban it does indeed 2019-06-13 10:58:25 bmbouter agreed, ty 2019-06-13 10:58:37 * ggainey is a HUGE fan of " focusing on the user experience and creating a safe one " :) 2019-06-13 10:58:37 bmbouter one more min! 2019-06-13 10:58:50 ttereshc ok let's schedule for the beginning of the next week? jsherrill should be back 2019-06-13 10:58:57 dkliban yes, lets do it 2019-06-13 10:59:06 bmbouter so w/ ^ idea this is way cool because you can actually have your tooling and systems use the P3 system, and maybe do some more sync's, download some content, whatever 2019-06-13 10:59:19 bmbouter and oh it's not perfectly right, just run the MP again! 2019-06-13 10:59:23 dkliban yep 2019-06-13 10:59:25 ggainey yeah 2019-06-13 10:59:28 bmbouter even while it's in production for other content types 2019-06-13 10:59:33 ggainey yeah 2019-06-13 10:59:42 dkliban yes, that is defintiely great 2019-06-13 10:59:43 bmbouter this is like rapid prototyping for your tooling around P2 -> P3 2019-06-13 10:59:49 bmbouter ty for organizing! 2019-06-13 10:59:56 x9c4 ttereshc: so it's 'justin-time' again? 2019-06-13 11:00:00 x9c4 ty 2019-06-13 11:00:00 ggainey we'll def want to pound on that kind-of usecase when it comes to testing/hardening 2019-06-13 11:00:01 dkliban LOL 2019-06-13 11:00:10 dkliban x9c4: that's right ... next week will be justin-time 2019-06-13 11:00:13 ggainey hehehe 2019-06-13 11:00:22 ggainey x9c4++ 2019-06-13 11:00:38 ggainey bah, needs moar pulpbot 2019-06-13 11:01:12 dkliban yeah ... ok .. i'll try to get pulpbot in here for the next meeting