As a wrap-up to our series, we are revisiting the episode that began the kickoff of the Art&&Code conference! In this re-release, AMT Lead Researcher, Hannah Brainard talks with David Lublin, a video artist, programmer, and performer based in Brooklyn, NY. Lublin is the co-owner of VIDVOX, a software company specializing in tools for real-time video mixing. In this podcast, they discuss the field of video instrumentalism, or ‘VJing,’ and how Lublin’s unique background as a mathematician and artist helped carve a path for his career. Since first checking out equipment at his university library, the pursuit of advancing his own artistic output has helped Lublin develop tools for fellow artists. Looking ahead, Lublin anticipates myriad opportunities for emerging technology in the field, including enhanced experience and audience connectedness through immersive augmented and virtual reality. This podcast was made in partnership with the Frank Ratchye Studio for Creative Inquiry.

Links:

• David Lublin’s Website 

• VIDVOX Website 

transcription

Hannah Brainard  00:11

Welcome to another episode of Tech in the arts, the podcast series of the arts management and technology laboratory. The goal of our podcast series is to exchange ideas, bring awareness and stay on top of the trends. My name is Hannah Brainerd, and I'm the lead researcher for AMTLab. Today, I'm joined by David Lubalin, a video artist, programmer and performer based in Brooklyn, New York. He's the co owner of a software company called VIDVOX, that makes tools for fellow video artists, some of the person that really just interesting work, I'll admit that I didn't know the term VJ before I started looking into your background. 

David Lublin  00:47

Yeah, 

Hannah Brainard  00:48

I was like, I knew what that was. But I didn't know it was this whole thing, this whole field. 

David Lublin  00:52

it's similar to the way that like, there was this period where like, VJing was just like your, you play radio records on the radio, and then it became this whole other thing. And VJing is kind of unknown, because went through this phase where was like you had your MTV VJs, who just showed music videos. And then there was this whole section of people who were like, no, no, we're similar to like the performing DJs. And we do live video stuff. But the term didn't really pick up. And there were people who, really who were in that scene who hated the term VJ, because they didn't want to be associated with this MTV thing. So they would come up with other terms like a video instrumentalist, or visualist, whatever kind of high art that they come up with that kind of applied to them.

Hannah Brainard  01:36

Is there a term that you prefer? 

David Lublin  01:38

I just go with VJ, it's fine. I think now that we've know that MTV doesn't actually play music videos anymore. I think the community has captured that term. Yeah.

Hannah Brainard  01:49

 That's great. So I'll just go ahead and dive into some of the questions here. So looking at your bio, I loved this line, you described yourself as an artist, programmer, performer and occasional mathematician, could you tell me a little bit more about what that means and how some of these interests come together in your work?

David Lublin  02:05

Sure. Well, when I was growing up, I was always a math nerd. When I was in high school, I was on the math team, and took like an every semester and of this extra math course. But when I went into college, I, for some reason, went into electrical engineering and computer system engineering, which I hated. And my senior year, I switched to being a full math major. And so math has always kind of been, I don't know, it gets in my blood a little bit, or my brain, or my heart. But I'm not a very good mathematician, I have friends who have like, full on PhDs, and they've done brilliant math things. And I consider myself an occasional mathematician, but it's sometimes in the course of writing software. I guess at this point, computer software design is evolved into like multiple different, it used to be this period where you go study computer science, and it covers everything that you have to do with making software. And now there's interface design, there's back end stuff, and then there's making algorithms for actual processing of things. So most of the time, I have to just do interface design or back in building. But every once in a while, I get to sit down and write an algorithm. And that is a very different programming process, even though that's what it ends up as ends up as code. But it starts as I need to do some math, maybe on paper, and figure out what's going on and then adapted into a code that gets optimized and things like that.

Hannah Brainard  03:26

Gotcha. Does that come in handy with your artistic work at all?

David Lublin  03:29

It does at times, probably one of my favorite, like one off performances that I've done before was kind of it was, you know, totally made up art stuff, like a little reality that I had made. But the presentation of it was like bar graphs and other kinds of data visualizations of made up completely made up data that was itself audio reactive, and it was used for like a live performance of the performance itself was his animated graphs that were changing over time, along with the music. That's interesting. 

Hannah Brainard  04:00

That's really cool. 

David Lublin  04:00

Yeah. Unfortunately, I did the implement like once and then just never really did it again.

Hannah Brainard  04:08

So just diving in a little bit more into your background, how you got started in all of this, I read that you started mixing video at a local bar, using video mixers, VCRs cameras and projectors. I imagine some of the technology that you use has changed over the years. So what are some of the software and hardware changes that you've seen?

David Lublin 04:27

It's interesting. So yeah, the background there was I was at RPI and all of my friends were in the arts department. So the arts department just like at CMU, I've watched here. You can they have all of these cameras and projectors and stuff. And as a student, you'd be like I'm working on a project and I need access to these. So we would my friends and I we were one of my friends at a local like Tuesday DJ night in this dive bar downtown that was across the street from where they lived. And we just started doing visuals there we plugged into the TVs at the bar and then we started taking out equipment from the equipment room and bringing it down. They're, and then eventually, like the professors were in the arts department became part of that scene as well. So they were doing weird performances and letting us take away too much equipment from the equipment room. And even at some point, like my roommates and I, we had our, you know, we were off campus went through a house party, and we like, made a list of like this liquid we want like for projectors, and we want these like lipstick cameras, and all this gear, and we brought it down to the committee, and they're like, why do you need this and we're like, well, we're throwing a party. And they were like, we can't  just give you this stuff for a party. And my roommate, who was one of my former business partners, and collaborators, artistically, still work on projects together. She basically made like a floorplan of our entire apartment, listed out where all of the gear was getting used with, well, we're going to take these lipstick cams, and we're going to like, strap them to the arms of the deejay. And those are going to go up onto the ceiling, you know, projection. And like we had, we had this and they were like, Okay, here's the equipment. This is this is what you guys are learning to do. This is what your crews learned to do. 

Hannah Brainard  06:00

So it's a project, not a party. 

David Lublin  06:02

It's a project at a party, right? It's a project party, even though it wasn't for class or anything like that, it was just so that's kind of how we got our start. But to get to your actual question of how the technology has changed, the biggest change has been from things moving from analog to digital, we were using analog video mixers. We loved the Panasonic WJ MX 30 and the WD mix 50. Yeah, just these with like that, like some pre baked, very simple effects, invert strove a Blur and Sharpen kind of things. But you can there on knobs, you could just turn them there was a giant cross fader for switching between your layers. And then there's a bunch of video inputs on the back. And now there are digital video mixers, and they do a lot of the very similar things. So in that regard, video mixers, yeah, come and go, they have changed in that way. And same thing with like capture devices that we were using. At that time, computers were like just at the point where they could start doing real time visuals we were working at, like standard definition, essentially 640 by 480, if we weren't lucky. And the thing that's really just improved, there is just computers can now do way more pixels. And I can you know all of the technical limitations where I was like, Oh, I can, I would need to set up two computers and have the moving into a video mixer so I could get what I wanted. And now everything can be done in one computer. But I still often think back to that setup. And if I can't, I still have two computers and the video mixer because I want that extra backup or there's always some other use for it if I don't want to, on the case where I can pack more, but it's nice when you want to pack like a laptop.

Hannah Brainard  07:37

So that kind of leads into some of your professional work with VIDVOX and VDMX, could you tell us a little bit more about the company, some of the features that it offers, and how being an artist has sort of helped you in that field?

David Lublin  07:50

Yeah, so the company was originally founded by this guy, Johnny DeKam in the late 90s, my friend and I took over the company from Johnny, at a certain point early 2004. And so Johnny had really came up with the idea for VDMX. And then we kind of took it more to the next level, he was kind of limited what he could do as a programmer, brilliant artist, great interface designer, but limited in terms of what his programming skills were. And he kind of pass it off to us. And we kind of took it a little bit further. We got into it though, because we were doing all of his live performances, we were writing our own little tools every we were trying to like every other week and make our different performance, it was totally different. And we were working in Max MSP at the time, usually, when we were writing our own software, and your video Max kind of came became the evolution of that because Max was great, but we had to was still programming and you still have to do it and build a blood infrastructure before we could get to something that you liked. And what you could do with the interface was limited in certain ways. So yeah, VDMX kind of evolved from that. It was like, Okay, we need a tool that's higher level than a programming language, but not so dumbed down that it's this fixed tool that is just for these cookie cutter kind of performances. And yeah, that's kind of how VDMX evolved. So it's, I usually describe it as like Photoshop for real time video, because you're still working with layers and groups of layers and composition modes of different Blend Modes. And you're putting effects on things. But instead of it being you apply an effect, and it's just rendered onto your pixels just once and you see the output. It's a stream of stream going through every layer and all of your effects have real time parameters that you can adjust. We also have you know, there's audio analysis that you can make things audio reactive. There's all kinds of ways of creating control data that automate your different parameters. You can set up cues if you want to have like something that's more of a structured performance that just kind of hit play on. And yeah, it's kind of one of like several tools in our field. And what I like about the field of real time video, and this is very similar to what I've noticed with music and a friend of mine, I'm kind of borrowing this from a friend of mine who's a teacher, music teacher, he was describing to his new students that for them, his goal was to teach them lots of different tools and lots of different techniques. And it was up to them to see which techniques and tools they liked, and combine them all to create their own unique style and equate it and I equate it to, if you're a painter, and you are limited to using canvas from one company, and only their paint brushes, you're very limited because you as a mixed, particularly mixed media artist, you want to try out all of these different tools, the best combinations of them for you and your style. And that's really important in kind of all fields of art, it seems like.

Hannah Brainard  10:42

So in some ways, it's like helped you create the art that you're interested in creating, you're able to pass that on to other artists to take on in their their own ways. 

David Lublin  10:50

Yeah. And so recognizing that they're going to use other tools. So interoperability is very important for us,

Hannah Brainard  10:56

Right. So speaking of some other tools, there's a lot of technology that's buzzing in the mainstream right now. AR VR AI all sorts of things. How do you see that fitting into the world of video instrumentalism or VJing? And what are you excited about? What are you skeptical about in that field?

David Lublin  11:16

So yeah, there's, there's a lot coming. When it comes to virtual reality and AR stuff and xr stuff. There's a lot of potential there. When I was teaching DJing. One of the assignments for the class was yesterday, music video, and using real time techniques, where you kind of record your output in a way that like a musician will do studio sessions, and then somebody will take them and edit them together. And the biggest part of what I was trying to teach them there was that when you're performing live visuals, everything around you is very immersive. There's big projections, the music is very loud and it's kind of it's very different experience than when you're watching a music video on a television screen, and the person kind of just get up and walk away. So you have to be far more engaging. And with AR and VR, I think there's a lot of opportunities for people who are visual artists, to create these worlds that are even more immersive, these experiences that are even more immersive. And engaging in that sort of way, something that he's not just going to get up and walk away from, if it doesn't have the right pacing or feels a little bit awkward. And it also kind of I think there's not just the personal experience of it, but the shared experience of it not just even in a single space, but allows for this interconnectedness of performance that are spread out distributed, not just another, you know, concert hall or something like that. I also know somebody who runs a planetarium in New York for like a used to run a planetarium is now designing a new planetarium, and has been trying to think about like, these new awesome live performances on the dome as part of what they do in the platform. So they've been talking about how they can use AR as part to enhance the experience of their planetarium so that people who are there can get kind of a personalized experience on top of the dome itself. So I think there's gonna be a lot of opportunities there. When it comes to like artificial intelligence and machine learning, there's a lot more in terms of like, the ethical concerns of what's going on there. In particular, like intellectual property. So there, I there's so many to comment on there. Yeah. What I'll talk about there is I did a project, maybe seven or eight years ago, 2006 ish, or 2016 ish, whether it was called TV helper, and it was a using a very, like rudimentary machine learning toolkit that those that company got bought by Google. And they work on much bigger things now, but at the time, they had this free open source, little, you know, you give it an image and it gives you back kind of image net results of like, here's what we think we see in the image, but really, not great because it was 2016. And those things have come a long way. Pretty good for that time, and free and most importantly it was fast. So I made a toolkit that watch live television, and let you as the user, pick what style you wanted the subtitles to be converted into. So we would watch television and you would turn to like, have like a MIDI controller like this where you'd have like, this slider would be for western style and this one would be for the style of the State of the Union and this one was the style of comedy and I pretty much like went and had a thing that parsed like television scripts I just in the way that ethically people are scraping things off the internet. I scraped ethically or unethically I scraped a whole bunch of scripts off of you know, some website that had television scripts on it, and loaded them into some algorithm that I made and connected that to things so you could watch television. But every once in a while he would say things that were just like, inappropriate or the juxtaposition of what it was watching were bad and I never found like a good way to have a system that self moderated itself. I know that that's a big part of what people do when they're building machine learning systems. But at the end of the day for this, there was no way that I could conceive of a machine understanding the all of the context behind who these celebrities might be, or what the situation was. So I went to a conference called Bot Summit and people were talking about how they made these Twitter bots and they would wake up every morning, and they have created all of this artwork for them. It was just posting, you know, out every hour, you got this image that it generated, and they were great. And every morning for me, it was I had to wake up and moderate my Twitter feed from my bot because it may have said something horrible, I may have lifted, watching the news and then made fun of a car accident or something like that. So my morning started off with deleting all of the things that were not great from my bot. And so I think that, at the end of the day, there should almost always be a human involved in the chain of that of creation and moderation at the end, I don't fully trust this. But yeah, but I do think that there are going to be some amazing art projects that come out of it from the people who do care about it in the right way. And I'm seeing people using the generative tools to create amazing things that they're using and VJ software already. And some of the kind of more simple algorithms are gonna be very useful, we're looking into like using some of the background removal stuff, because that you can do real time now. And rather than having to use the green screen or something like that, you can just use the same algorithm that Apple has for your iPhone, you can just drag somebody out of an image now. And you can use that it works in near real time. So we're looking to adding some of that stuff where it's kind of like a moving piece in a larger project that you might do,

Hannah Brainard  16:45

Rght? So a tool to assist in the creation process, not a replacement of the human asset.

David Lublin  16:50

I like the term assisted intelligence as opposed to augmented intelligence, as opposed to artificial intelligence.

Hannah Brainard  16:56

I like that. So just to conclude, my last question for you, could you tell us a little bit about some of your recent projects or other projects that you've seen that you're really excited about?

David Lublin  17:07

Since the pandemic I have not had, I feel like all my creativity is flowing out of my body. These days, I get most of my kicks out of a friend will have a project? And they'll be like, can you help with this last 5% on a thing? And I'll be like, yes, I would love to not have to think creatively, I can just come in and help with some technical aspects or give you some advice. And you're just gonna direct me on what you want me to do. And so I've been really enjoying that I have one of my good friends has been really getting into doing analog synthesizers, and making their own of those. So I think that will kind of maybe be my next leap of they're all of my friends who are in that area are like, Hey, you should get you know, make your own Eurorack and analog synthesizer stuff, and then start doing make some video modules for that. I'm like, Okay, maybe we'll maybe after I get like the current will fit box wise, we're just working on our next major release, which is a new opportunity, exciting update, because it's mostly just switching from Legacy OpenGL code and metal code, which a lot of companies are in the position of having to do now. And it's not not very exciting to talk about. We're just at the point where we're like optimizing stuff, it's getting back to like, how has technology changed more than the divide from analog to digital, it's that even developing software, you have to do all of this maintenance year after year of just keeping up to date with the latest technologies, even when you're not adding new functionality sometimes, but this is, I guess, that's exciting for us. Because once we get that, over that hump, to being in metal and Vulcan, we can take advantage of all of that stuff. So all of the stuff that Apple now provides in the vision libraries will be easier for us to access. So that's like, over the horizon for us. But we still have to deal with a lot of technical debt right now, which is not something that they teach you about so much when you're learning in college. You don't teach about the amount of technical debt that you take on, or that when you go to work for another company, and they're like, you're really excited to work on something new. And they're like, Well, we have all this technical debt, we have all this stuff that's fixed in place and needs to be updated or maintained and it's not sexy work at all.

Hannah Brainard  19:08

Like I thought this was gonna be fun. Yeah. Exciting. And, and all that's coming. Yes. Right. As soon as the Yeah.

David Lublin  19:14

But I when I see my friends were doing analog synthesizer stuff. It really reminds me of the early earlier days where we were just plugging stuff in and having a really good time just messing around and making weird sounds and video.

Hannah Brainard  19:26

So kind of a step back,

David Lublin  19:28

step in another direction for a little while, but there's enough overlap where it fits for me.

Hannah Brainard  19:33

That's great. Are there any other projects that you're excited about in the field or

David Lublin 19:38

not at the moment? No, I wish there were but we were I was talking with somebody else yesterday about how it seems like there'll be maybe fads or like periods that there was this period where like everyone was doing projection mapping for a while that was the big thing. And now it's become kind of this, like solved problem. People know how to do projection mapping and all the big projection mapping problem, projects that were like just done for vanity have really dried up. People aren't doing those as many as often. And now it seems like the next role of that is going to be AI and ML stuff and just kind of coming off the tip. But again, those things aren't quite at the point where they can do fully real time stuff. So it hasn't really made it into the live visual scene totally yet.

Hannah Brainard  20:21

Gotcha. So something to look out for. 

David Lublin  20:23

Yeah. I think in the next year or two, you'll start things are getting there. Okay.

Hannah Brainard  20:29

That's great. Well, thank you so much for your time today. Thanks for being at CMU for this program. This is this is really exciting.

David Lublin  20:35

Thank you for the wonderful interview.

Hannah Brainard  20:37

Thank you for listening to tech in the arts. Be on the lookout for new episodes coming to you very soon. If you found this episode, informative, educational or inspirational, be sure to send this to another arts or tech aficionado in your life. You can let us know what you think of the podcast by visiting our website amt-lab.org That's AMT-Lab.org or you can email us at amtlabcmu@gmail.com. Follow us on Instagram @techinthearts or Facebook and LinkedIn @Arts Management and Technology Laboratory. We'll see you next time.