Moderator (MDR): MDR-lisa

Guest-Speaker (SPK): SPK-jdr

MDR-lisa: Hi, everyone. I'm Lisa Stapleton, editorial director of the JDC, and I'll be moderating the Code Clinic today. We have a skillful coder here, David Rivas (login: jdr), whose expertise is in multimedia and Java^TM. He is the author of the Java Sound specification and wrote the code for the Java Multimedia Framework.

David, I understand that you just released some new documents relating to Java multimedia. While we're waiting for the first code questions, would you like to talk about them?

SPK-jdr: At Siggraph this year we announced the beginning of work on the new Java Sound API. Java Sound takes a very high-powered audio-rendering engine and puts it together with a new API to provide really high quality audio output with sound synthesis functionality.

The Java Media Framework (JMF) is a collection of APIs for doing time-based multimedia, all in Java. The key idea in JMF is scaleability across a large number of protocols and data formats. We've been developing this in conjungtion with SGI and Intel, and are releasing the 1.0 version of the spec at the end of this week. We're really excited about this one.

abennett: I've got a GIF in a ZIP file. How easy is it to stream in the bytes and create an image object?

SPK-jdr: There are two questions here: 1.) How easy is it to decode the ZIP and get an image out of it, and 2.) How can I stream the data?

abennett: Yes, that's right. I'm currently thinking of an uncompressed ZIP file to display images, but certainly it can apply to a compressed ZIP file. I can get the ZipEntry, that's fairly easy. My problem is basically getting the image out of the ZIP and creating it with createImage.

SPK-jdr: Have you been using the ZIP packages?

abennett: To some respect. I haven't pulled a stream of bytes out of them quite yet. java.awt.image.MemoryImageSource seems to be the way to go after I can get it out of the ZIP.

SPK-jdr: You should create a ZipInputStream, move the stream
to the appropriate entry, and read from that.

chonok: I beleive that Sun recently licensed the HeadSpace Audio Engine. Will Java be able to "play" RMF music soon(ish)?

SPK-jdr: Absolutely. The next release of the JDK^TM will support RMF files, through the (yes, very limited) AudioClip interface. Additionally, AudioClips can be constructed by applications.

GordonGaar: Is there support for in 1.1.3? I have not seen examples of their usage in the current doc.

Will there be an effort in future releases to make the use of animation and sound differences in applets and applications more consistent? In other words, will the multimedia framework be supported to a greater degree for applications in the future?

SPK-jdr: I don't think the animated GIF stuff made it into 1.1.3. It might be waiting for the next major release.

There is a large effort here to incorporate synchronized audio, video, and animation into Java. The JMF and the Java Sound APIs are a big part of that. We've also been doing some work on multiplane animation and scripting APIs.

What are the key features you think you need to see here?

abennett: Will the JMF have support for streaming audio and video sources?

SPK-jdr: Yes. The center of design is support for media that either comes
across the Net, or comes from a local machine. We have an architecture that actually allows the protocol handlers to come bundled along with the media. We will have implementations supporting RTP protocols. Right now we are involved with a couple of the streaming players, working to get their proprietary protocols supported by JMF.

We'll also bundle along support for playing QuickTime files and AVI files, as well as the most popular audio and video decoders.

JoeSam: I just wanted to agree with Gordon that it would be great to be consistent in the capabilities of applications and applets insofar as possible.

SPK-jdr: Right. Sorry about missing that in Gordon's message. There are two interesting places to worry about compatibility--between applets and applications, and between different APIs.

Security issues not withstanding, we are working pretty hard to make sure that applets are "first-class citizens."

With respect to APIs working together; we are now all one media group. For example, JMF Video Players are using the latest 2D functionality to get enhanced rendering. Another example is making sure that our 3D work is rolled into the audio support engine in the form of shared, 3D, audio APIs.

GordonGaar: I would like to be able to use animated GIFs with a simple start, stop, and restart, so that GIFs will be at the begining on request.

SPK-jdr: Or even position to a particular frame, right? We've got that nailed for generic playbacks in JMF. But I don't think that has migrated up to the Image API.

abennett: Will the various players initially have options to allow "fallback" to lower quality resolutions of the streamed data dynamically, in case of the net.lag monster? Or will that come later?

SPK-jdr: Not to be a weasel about this, but yes and no. To do this properly you need support in the protocol that is delivering the bits to the player. We probably will not ship one of these as part of the first implementation, however we are working with a small streaming company that has the kind of technology available. We are doing everything we can to make it easy for them to ship a JMF player that does this. Since the protocol-handling piece of a JMF Player can usually be written entirely in Java, we think we'll see this functionality quite soon in many players.

MDR-lisa: David, do you have a URL for some of the docs that you've talked about?

SPK-jdr: Sure. The place to look for all things multimedia on the java site is: http://java.sun.com/products/java-media/index.html

We are just about to make the 1.0 JMF specs public, so look for them at the end of this week. Also, we expect to have a Java Sound API up for public review fairly soon. I can't urge you all strongly enough to read the specifications, comment on them, and join in the discussions on the email aliases. These public reviews are really important to us.

MDR-lisa: I have a question for our audience. Has anyone been using Sun.audio stuff? David points out that some of that earlier stuff will change, and wonders whether you want to know what's different.

abennett: No Sun.audio.* here. I've been expecting them to change. I've stuck with AUs in the meantime.

SPK-jdr: I'm sorry about that. We've definetly fixed the AU thing--WAVE, AIFF, and AUs are all supported. We've also got a fairly low-level API to allow you to render audio bits from memory--finally!

Most of the sun.audio.* stuff--while we'll continue to support it as long as it makes sense--will be completely replaced with Java Sound.

Additionally, we're adding synthesizer interfaces so people can add fairly complicated sounds to applications and applets. In some sense, you can think of the JavaSoundized VM as a complete electronic musical instrument that can be controlled by the VM.

abennett: What I meant was that I've expected the Sun.* classes to mature as Java matures--not as a slam on them, and no harm. Simple AUs have met my needs to this point in time, but the concept of rendering audio bits from memory (sounds like my image problem, he he) sounds particularly cool.

SPK-jdr: Right. We've gotten quite a lot of positive feedback from early access folks on this. There are a lot of musicians and creative content providers that think this will let them do things that they haven't been able to do before--like provide sounds that react to the changing nature of the environment, or simply added music that changes as the web page changes.

jacquesl: Are there any mail listservers/newsgroups I can join to find out more about this Java Sound API?

SPK-jdr: Sure. If you go to the URL I sent out before and click on the Java Sound pages, you will find instructions on subscribing to the Java Sound and the Java Media Framework pages.

abennett: So given specs and their implementation, when do you think one would be able to make a Java application that simulates a simple keyboard in terms of functionality? (I'm thinking of things like RubberDuck, that Roland TB-303 emulator and stuff like them that are currently Win32-based.)

SPK-jdr: With Java Sound you can easily do the things that you just described. Essentially there is a fairly sophisticated wave table synthesizer, all in software available through the API. I'm not supposed to say what the release dates are, but I would expect to see a synthesizer emulator application by end of year.

jacquesl: I've worked with MIDI low-level stuff doing ActiveX and Java works, but it is not very portable. (I wish Sun had something on MIDI in Java.)

SPK-jdr: Java Sound has a built-in MIDI sequencer, and MIDI control capablility in the synth. So it can play back sequenced MIDI files and also let an application send MIDI commands to the synth. And it's portable!

MDR-lisa: So what new features are we getting with the Java Media Framework?

SPK-jdr: The initial release of JMF adds the ability to put playback of audio/video streams and files right into applications and applets. The Player abstraction gives control over the media and provides the AWT components that the media data is rendered into. For example, you can add a QuickTime movie right into the application, put the Video Window wherever makes sense in your app, and either control the display in the application or provide a GUI control for controling the media.

This is designed to be as painless as possible without giving up the functionality that more complicated apps require. One of the key points we try and make about JMF is that it is all about synchronization. Synching the media of course, but also synching the media to the reset of the appliction.

There's a guy who is doing a demo for JIBE that has a bunch of MPEG files that tell an interactive story that has a bunch of points where, depending on what the user does, the video gets branched to a different place. He's got the whole thing working in JMF. We were all very impressed.

jacquesl: Can you give us a Java release (ie. 1.2 or 1.3)? This sounds really exciting.

SPK-jdr: Here is what I can say: the Java Sound piece 1.2 will have the new engine in it. We are tying the rest of our schedules to the 1.2 release cycle. The full API for Java Sound should be available in the same time frame, but we may not actually bundle it in 1.2. The reasoning is complicated and circuitous, but there it is.

JMF will have a public release also in the same time frame as 1.2. There is an internal discussion going on as to how best to release it. Due to size constraints, there is a question over which release the JMF will go in, and how it will be packaged. This is all up in the air right now.

hcobb: Which of the "universal" database vendors have announced support for JMF in their multimedia frameworks?

SPK-jdr: Nobody has made an announcement yet. Fortunately the database piece is not quite so tied to the actual media delivery vehicle, which is what JMF is all about.

For example, there have been a few announcements about support for RealVideo from Progressive Networks. We have spoken with Progressive about a JMF player, and have had an extremely positive response.

But the point is that it shouldn't matter from a database perspective. The service providers change the media delivery mechanisms at will, and we expect JMF Players to be available for all of the popular delivery mechanisms. Because so much of a Player can be written in Java, we are expecting the Player to come right along with the media data in many cases.

jacquesl: http://java.sun.com/products/java-media/index.html only highlights FAQs. Is there anywhere I can get more on the Java Sound API, especially with the general MIDI stuff? I'd really like to get my hands on this as soon as possible, any morsels are more than welcome.

SPK-jdr: We've only just announced the work on the API. While there is a bunch work going on internally, we are contractually obligated to give them first crack at review. However, by subscribing to the mail lists, the minute something is available, you'll get a notice. I would expect that you'll be able to see the API for review in about a month or so. I really want to encourage you to subscribe to the list if you care what the API looks like. (By the way, the folks to which we are contractually obligated, give first review access and are Java licensees too.)

JoeSam: To get back to the Code Clinic, any tips/techniques on the "best" (fastest, most efficient) methods of loading and displaying images? Same for animation? Any traps you see programmers typically falling into in this area?

SPK-jdr: One of the problems we seem to see a lot of has to do with incorrect usage of threads for synchronization. There are a lot of people who rely on timers that wait in a thread, which when they wake up move something along. Not only is this inefficient, it doesn't provide very good synch.

The fix is correct use of notify/wait. The key thing to remember here is that in most animation you have an action that should produce a reasonable reaction. Use this action to notify a thread waiting to produce an event, don't spin in a sleep loop and then check a semaphore.

Also, we find that a lot of people don't make reasonable use of the ImageObserver model for discovering when images really are available. This is a performance and redraw overhead associated with constantly adding a little bit of an image to a screen. By waiting for the image to complete, and then notifying a waiting thread that the image is ready to display; you not only decrease the flicker associated with the partially displayed image, but you also decrease the time it takes to get the image.

Finally, if you're downloading a bunch of images--depending on the size of the connection--it may make sense to limit the number of images you are going to download to only one or two at a time. Essentially it's a thrashing issue. The overhead associated with managing multiple images starts to significantly affect the download time, so that in some cases you'll see faster performance by just downloading one or two images at a time.

abennett: For many images, wouldn't it make sense to put them all in a ZIP or JAR and download them with the rest of the applet code?

SPK-jdr: Absolutely. It depends on what level of interactivity you want to provide with respect to images--if the images are being generated on the fly, and if you want to hit the whole applet with the JAR download.

MDR-lisa: Several of you have asked where to go for more info. There are some interesting transcripts from JavaOne^SM on 2D, 3D, Sound, and the JMF, which you might want to check out, and the JDC has several articles about these topics under "Read About Java" and "Java in Depth."

chonok: Just a thought on JMF--if the way that AWT has revolutionized GUIs is anything to go by, I think we could see major changes in general computing once JMF is released. The possibilities seem endless. :-)

MDR-lisa: Thanks everyone for attending, and thanks to David for participating. Be sure to check the schedule for more Code Clinics, Java Live, and the JavaBeans^TM Short Course Office Hours.