Community Information Needs

Facilitator: Mayur Patel, John S. and
James L. Knight Foundation

Scribe: LuAnn Lovlin, The Winnipeg Foundation

Information and Digital Media

Citizens require relevant, contextual and authentic information to run their lives and to manage their communities. Information can also be a creative and powerful tool for achieving social
change. But how do we evaluate whether our media and communications projects
make a difference in the world, how do we make our projects more effective and
how can we learn lessons for the future? These questions formed the main part
of the discussion during the breakout session.

Information and communications projects
cover a wide variety of activities, including TV, radio and film,
telecommunications-based projects, broadcast campaigns and web-based projects.
In evaluating the progress and impact of these activities, our approach should
be guided with the end user in mind (ordinary citizens, communities,
organizations or policymakers); be inclusive of all participants and contain
credible and accurate information.

Our Challenge

There are several practical difficulties
involved in evaluating the impact of information and communications projects.

• The goal of these projects, which often
  includes outcomes such as greater public awareness, increased dialogue and
  empowerment are difficult to measure objectively, especially when compared
  to projects that provide direct services, such as education, health and
  housing.
• It can be difficult to define a specific
  target audience for a media initiative (some projects are geographically
  defined, whereas others involve wider forms of broadcasting).
• Social change often happens slowly, which
  makes it hard to capture a project’s impact over a short period of time.

Understanding Social Change

During the breakout session, participants
discussed how to adapt traditional logic models to the area of information and
digital media, namely: a) how to conceptualise the linkages between outputs,
intermediate and final outcomes, and b) what leading indicators reveal whether
a project is on track to achieve success. Depending on the particular project
these might include:

• Outputs: the launch of a website; adoption and utilization rates; levels and depth
  of distribution within a community; the establishment of strategic
  partnerships with other organizations, e.g. newspaper, community centers
  etc.

• Intermediate Outcomes: increased dialogue on a particular issue within a community; analyzing
  site traffic peaks during times when local issues are discussed; greater
  collaboration between different members of a community; higher levels of
  deliberation.

• Final Outcomes: changes in public awareness of local issues; changes in
  individual attitudes or behaviour; greater social inclusion; the formation
  of new advocacy related efforts or organizational practices; changes in local
  state policies.

Tracking Online Behavior

The discussion about outputs and activities
focused on the need to capture online patterns of behavior. Increasingly media
projects make use of web-based technologies and social media tools, which
provide us with the opportunity to track a variety of indicators of people’s
behavior. These include:

• The number of unique visitors, frequency
  of visits, time spent on site, depth of visits; referring URLs; natural
  search results; number of registered users; responses from the blogging
  community; technorati authority ranking; number of RSS subscribers;
  conversions of visitors to contributors on the site etc.

Organizations can use several free online
tools and statistical packages to capture and analyse these indicators,
including Google Analytics, Woopra (especially useful for less trafficked
websites, such as blogs) and CrazyEgg, among others. However, measuring
clickstream data does not capture people’s levels of engagement and does not
provide insights into why members of a community are using a website, are they
completing what they set out to do and are they satisfied? For this qualitative
data is important. Here a few tools have emerged, which allow organizations to
survey their website visitors, e.g. 4Q. The next step, understanding whether,
and how, levels of online engagement lead to offline action in communities
remains an ongoing challenge.

What have we learned so far?

At the close of the session, four
principles were highlighted to guide evaluations of information and
communications projects:

1. Capture real time feedback – increasingly information projects have an online component,
   involving the use of social media tools and web-based technologies. This
   provides us with an opportunity to analyze the formation of new
   relationships, conversations and networks, provided that we’re actively
   looking out for these activities.

2. Recognise the importance of outputs­­ - social change takes a long time, and so we should be sure
   to value a project’s ability to achieve particular milestones. These
   outputs should be thought of as set of leading indicators of a project’s
   potential success in the long run. In setting these outputs, think about
   what would reveal whether a project was on track to have the greatest
   possible chance of success.

3. Flexibility is key – given the rapid pace of technological change, the field of
   information creation, sharing and gathering is constantly evolving. This
   makes it difficult for projects to anticipate potential obstacles.
   Organizations and foundations should be cognisant of these challenges.