Prototype Evaluation 7/27/16

Week 5: Prototype Evaluation 

This week, we first refined our 3D model in Unity by replacing the floor plan with a measurement grid and adding extra walls in the virtual space for participants to manipulate.

Final model for this week's prototype evaluation

Our research questions remain unchanged from the previous week:

• What aspects of our design help collaborators establish common ground efficiently?
• How does our design differ from traditional mediums in terms of interactions?
• What elements of traditional mediums “map” well to a virtual environment?
• What challenges do clients face when using our solution?

We decided that for this round of evaluation, we might not be able to answer the first question since it requires synchronous communication during the design process to evaluate effectively, and our prototype currently does not support networking.  So, we temporarily removed the question from this first round and focused on how our solution helps clients to better understand and communicate spatial concepts to designers. 

For the evaluation, we focused on three major research goals:

• Understand how clients will use our solution to illustrate their ideas
• Identify possible challenges and the learning curve of our solution for further refinement
• Evaluate the core values that our solution brings to clients as well as its advantages and disadvantages

Evaluation Overview

Based on our research goals, we determined the prototype evaluation focus:

• Understand how changing viewpoints between small-scale and full-scale might facilitate clients’ spatial understanding.
• Understand how clients utilize and value object manipulation and 3D sketching functions for changing the spatial layout. 
• Compare the client experience between 2D floor plans and 3D virtual space to identify advantages and disadvantages.

Evaluation Process

We designed a test protocol based on two conditions. Each session evaluated a single participant and took about one hour. Participants performed the same four tasks with both our solution and pen/paper. The order of conditions was counterbalanced. At the end, participants were interviewed about their experience using both mediums.

Evaluation Approach

Recruitment

After the model refinement, we started our recruitment of participants. To ensure the participant group could represent our target users which were project owners/investors, our criteria were that the participants should have a non-architecture background and have experience working with architects/interior designers or experience designing their home/workplace.

Scenario-based

The participants were described a scenario where they are planning a 1B1B one-floor home and just received a design concept from the architect. 

Participants were asked to perform four tasks based on the scenario: 

1. Removing an extra room
2. Separating the living room space
3. Increasing the amount of natural lighting
4. Further decoration of the whole space

Two conditions

Participants completed a series of tasks in both conditions (traditional floorplan & our solution) which were counterbalanced across participants.  This allowed us to compare how their experiences and understanding differed between traditional markup methods and our solution.

User Testing

We recruited four participants based on a set of criteria that helped us approximate the target profile of non-expert clients that might be working with designers to plan a space.  We also accounted for other confounding variables such as gender differences and VR experience.

• 4 participants (2 male, 2 female)
• Non-architecture background
• Has experience planning home or workplace
• Little to no previous VR experience

The testing took place at Startup Hall at the Univesity of Washington. We set up a room-scale VR system and ran one session on Friday afternoon and three sessions on Tuesday.

We used video cameras as our primary method of capture. In the VR task, we recorded the projected screen together with the participant due to the importance of body language in understanding their intentions. The audio was also recorded throughout each session since participants were asked to think aloud as they completed the tasks.

Findings and Implications

Key finding #1

Participants had more satisfying and reactive experiences with the manipulation function in virtual space because it provides instant feedback on the spatial change.

Implication

The real-time feedback and visualization help users feel more satisfied with their design decisions, which could be a core value of our solution.  We should consider the effect of gratification on client involvement.

Key finding #2

Participants were more willing to experiment with different changes and specify the design when they are in VR compared to using pen and paper. Participants found that they easily got lost when sketching on paper due to the permanent nature of the medium.

Implication

Our solution is valuable for encouraging user involvement due to the ease of experiencing changes without the need to commit to permanent modifications.  We should continue to facilitate this kind of interaction and redesign the sketching function to reflect this value.

Key finding #3

The flexibility of jumping between small-scale full-scale views increased participants’ spatial awareness.  In particular, participants used the small-scale view for an overview of the spatial layout much like a traditional floor plan.

Implication

The intuitiveness of the small-scale view transfers from traditional floor plans since it allows users to see the overview of the designed space.  Perhaps we should emphasize this view over the full-scale and refine it to carry over more valuable aspects from floor plans.

Key finding #4

The sense of measurement was more subjective to our participant group. In our prototype, they measured the size and distance (when manipulating objects) relative to other elements in the space instead of referring to the measurement grid.
However, most participants expressed a desire for precise positioning and explicit measurement visualizations.

Implication

Clients considered our solution more as a tool to experience layout and design changes, but it is possible that designers have a different concern when looking at the space. We should investigate more on the need for precise measurement.

Refinements

Based on user's behavior and their comments, we found several user experience and usability issues that we should take into consideration and go through a refinement.

User experience

• Participants found it challenging to sketch 3D concepts precisely to present their ideas.
• Participants found that it was hard for them to measure distance when moving walls and furniture.
• Participants could not tell whether the furniture was the right size
• Participants often made changes while in small-scale view. However, many reported dizziness or neck strain from looking down to perform tasks.

Based on the evaluation, we proposed following solutions to refine the user experience.

• We plan to provide an indicator or automated snap-to-surface function to help users with sketching.
• Dynamic measurements that will pop-out showing the exact distance while moving an object. 
• When pointing to furniture, users can see the dimension information.
• We are lifting the small-scale model up based on participant height and making it adjustable so that they no longer need to keep looking down.

Usability

• Participants sometimes lost their sense of direction and location in VR, especially after teleportation.
• The controller button mapping was often difficult to remember, causing participants to trigger unintended actions.
• It was sometimes difficult for participants to make small manipulations with precision (e.g. to the walls and furniture).

Changes we plan to make to improve our prototype:

• Add reference objects or marks into the space to help participants locate themselves.
• Add labels to the virtual controller model for reference and streamline button interaction.
• Add a snapping function to help align objects to other objects automatically or tweak manipulation interaction.

Next Steps

Evaluate the common ground 

We will present the virtual markup to designers via POV screen recordings of participant sessions and let them interpret the participants’ design intentions. We will also revisit video recordings and listen to participants’ thought processes so we can evaluate the accuracy of interpretation. This will help us determine if mutual understanding is reached and what parts of our solution were effective.

Refine the prototype

We will select the most significant usability problems relative to our topic and refine our prototype for the second round of testing.

Final Takeaways 

Despite our initial assumptions, the 3D sketching function was not as intuitive as we thought.  While most participants thought that the sketching tool was valuable for labeling and marking up the space, it was often challenging to use.

Another assumption was that our solution would provide an increased sense of scale. However, it was compromised by technical limitations in our prototype. It would be ideal to maximize the real-size feeling of the full-scale view by increasing the freedom for participants to walk around in the space so they would not have to rely on the teleport function quite as often.

Reflection

Our evaluation helped validate our design solution as participants expressed enthusiasm for the value of real-time 3D markup. Participants’ overall positive feedback on the manipulation function made us realize how instant visual feedback can help participants experiment with and understand spatial concepts to inform their decisions.  While rough, our initial prototype was functional enough to be able to help us evaluate our research questions and reveal specific improvements for refinement.  We are confident moving forward with our solution in the final weeks.

Prototyping 07/20/16

Week 3 & 4: Prototyping

During the last two weeks, we were focused on detailing the interaction flows of our final concept and planning our prototype evaluation. We made our decision to focus on the asynchronous collaboration experience between clients and designers based on two reasons. First, asynchronous communication is the most common channel in remote collaboration context. Second, it is easier to implement into prototype due to time and technical constrains.

The record of the first prototyping meeting.

Interaction Flow

To understand how our target user will use our solution, we created an interaction flowchart to show all possible interactions. 

The whole concept contains three main interaction flows, which are synchronous collaboration in the same virtual space, synchronous collaboration in separate spaces, and asynchronous collaboration between clients and designers. The asynchronous collaboration experience will be the main focus for our prototyping evaluation. 

Although we are still in the process of finalizing our research protocol, we have identified the big picture and direction for our prototype testing tasks and evaluation.

Prototyping Overview

After consulted with instructors, we began to discuss about which technology should be used to build our solution. In the meanwhile, we also brainstormed some research questions that we would like to answer.

Meeting notes from the second meeting

During our third meeting, we further put down all the requirements and functions that should be included in our prototype and split the work so we could start working. 

The second meeting mainly focused on two parts, which technology should we use, and who should be our participants.Since we are going to focus on client's needs, we determined that our target participants should be:

• Non-architecture background
• Has experience decorating home/workspace
• On the market for new houses or apartments (either currently or in the past year)
• Has experience working with architects or interior designers to design a space

And we decided that our prototype should have three basic functions.

• 3D sketching
• Object manipulation (translation and rotation)
• Full-scale and small-scale viewpoints

Further details of the prototype was discussed during our third meeting. We discussed and listed our all the details including where to get the resources, how will each function being used, how should we test our prototype, and how our prototype will answer our research questions.

Research Question Formation

On Wednesday and Thursday, we had a meeting with the instructors and sponsors. We presented our ideas and collected feedback, some feedback we got included

• Focus on big picture rather than interaction details
• Emphasize synchronous collaboration
• We should consider Wizard of Oz method

The meeting helped us revisiting our design concept and forced us to think about what is our design hypothesis, and how are we going to proof that our solution is better than the traditional method.

Design Concept

Based on the concept we discussed during the concept refinement phase, our solution is a model-based interface shared between remote collaborators that facilitates dialog and spatial understanding through the establishment of common ground.

Furthermore, stakeholders share the same virtual environment and experience different viewpoints to gain a full sense of scale while allowing real-time manipulation of the model.

Design hypothesis

Our design provides clients with 3D markup tools in a full-scale virtual space to help them better understand and communicate spatial concepts to designers.

Research Questions:

• What aspects of our design help collaborators establish common ground efficiently?
• How does our design differ from traditional mediums in terms of interactions?
• What elements of traditional mediums “map” well to a virtual environment?
• What challenges do clients face when using our solution?

Research Goals:

• Understand how clients will use our solution to illustrate their idea
• Understand possible challenges and learning curve of our solution
• Understand how successfully the solution helps clients communicate their design intentions to designers

Evaluation Plan

After we got our design hypothesis and research questions ready, we started to put down more details of how are we going to test the prototype.

Research Protocol:

1.Design with pen and paper on 2D medium

An empty floor plan and a list of requirements will be provided to the participant, and they will be asked to leave markup on the spatial arrangement on paper.

2. Design in VR

The client will first go through a tutorial to learn the basic controls in VR, then be placed in an empty virtual room with a list of requirements. They can sketch in 3D, manipulate objects, and jump between viewpoints to communicate their ideas.

3. Debrief and evaluation

The client will be asked to compare their experiences between using traditional pen & paper on 2D floor plan and using our solution. We want to understand what tools they prefer and why as well as any challenges they encountered during the session.

4. Expert evaluation

The client’s markup will be shown to the designer. The designer will be asked to evaluate the two different mediums to help us understand what aspects of our solution are the most effective and whether our solution helps the formation of common ground.

Documentation Methods: 

We plan to record video of the participants as well as screen recordings to capture the complete point of view in virtual space. We will also take notes during the sessions to record any particularly interesting insights.

Materials: 

For the prototype, we have made a floor plan of a 1B1B one-floor interior, and also built the virtual environment of the same interior in Unity. Both of them will be used for the testing in order to compare the 2D and 3D methods.

Prototype Demo

Finally, we had our pre-alpha version prototype ready. Although it is still pretty rough, we got all the functions we need for testing. The next step is to run some pilot study and start recruiting participants to evaluate our solution.

Reflection

Our prototyping phase began with extensive discussion to figure out what part of our solution should be built and evaluated.  While this was difficult to determine initially, our time constraints and interaction model made it clear that we should focus on asynchronous collaboration for our initial prototype.  Building our prototype in Unity was relatively straightforward and we were able to build the virtual environment and implement the basic functionality over a weekend.  Getting this first iteration completed also made it clear what features still needed to be added in preparation for our user tests.  Instructor and peer feedback on our presentation was positive and we are confident in moving forward with our prototype.

Concept Refinement 7/6/16

Week 2: Concept Refinement

In week 2, we further narrowed down our ideation to one concept. We revisited the problem space and decided that the core value of our solution is to "help form common ground between designers and project owners." Based on this value and our previous investigation, we created a concept image, storyboard, interaction model, and system diagram to help us convey the idea.

Refined Project Focus

Facilitate client contribution in early stages of the spatial design process to help the formation of common ground between designers and project owners.

Background

Sharing common ground (i.e. mutual knowledge, beliefs, and assumptions) is essential for communication between collaborators (Clark, H.H. & Brennan, S. E., 1991). However, architects and project owners working in a remote context face two major challenges.

First, it can be difficult for collaborators without spatial design backgrounds to fully understand design intentions. Differences of expertise in project owners make it easy to misunderstand 2D artifacts such as floor plans and images, or how the final design might look after being built. This kind of miscommunication can be extremely costly in the long run.

The second challenge is ensuring that people are referring to the same objects or locations during discussions. With gesture and gaze, referential communication diminishes the effort needed for mutual understanding during face-to-face communication. However, in a remote context, people might not be able to rely on these non-verbal cues to establish common ground. Remote collaborators need to spend more effort to get on the same page, resulting in inefficiency and possible misunderstanding.

Concept Overview

A model-based interface shared between remote collaborators that facilitates dialog and spatial understanding through the establishment of common ground. Stakeholders share the same virtual environment and experience asymmetrical viewpoints to gain a full sense of scale while allowing real-time manipulation of the model. 

Target User

Project owners and investors 

Architects and interior designers

Problem Space

Most project owners or investors have a difficult time understanding space and sense of scale from 2D images of design concepts such as floor plans. On the other hand, it is also difficult for designers to fully capture client needs and expectations by referring solely on text-based markup without more in-depth social cues. The communication of design feedback through traditional channels such as email also causes considerable delays to the project schedule.

Value

Remote collaborators can establish common ground by breaking technical barriers and utilizing referential communication. The solution facilitates the design process by presenting a modular experience with tools to help visualize non-verbal cues.

Interaction Model

The model/space itself stands as the common interface for both designers and project owners. Designers manipulate the model in the small scale while project owners experience the updated changes in the full scale. Designers can understand project owners’ attention and needs by sharing their gaze, reading their gesture and listening to their voice.

System Diagram

The system diagram maps out the stakeholders involved in the design process and their relationship with other components in the system. Steps 1 to 7 indicate the general design process and show how the solution is integrated into the system.

Reflection

Overall, we feel confident about our design direction and will push forward with prototyping the concept.  After spending a long time debating as a group on whether to present several concept variations or just a single concept, we found that the single concept stood better on its own than several incomplete variations.  Instructor and peer feedback brought up the issue of asynchronous vs. synchronous collaboration and how our solution might continue to function in the project workflow.  As we begin prototyping, we plan to pick a smaller function of the interaction to build in order to evaluate it.